Turn off color for downstairs dump Removing existing region "/var/tmp/test_repair" Creating region directory at: /var/tmp/test_repair Creating dsc directory at: /tmp/test_live_repair/test_repair_dsc Downstairs region 0 created at /var/tmp/test_repair/8810 in 0.30869785 Downstairs region 1 created at /var/tmp/test_repair/8820 in 0.019504365 Downstairs region 2 created at /var/tmp/test_repair/8830 in 0.01849585 Created 3 regions /var/tmp/bins/crutest with -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 1201 1202 1203 Dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.240251448Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.308821419Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.311143366Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.311253782Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.31127487Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.311297817Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206} {"msg":"Crucible fd0b18fe-644d-4c7c-a791-5a4c8cf83ba3 has session id: 8a365dcc-cd88-400e-ba8d-c6dda593006d","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.312431184Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"Upstairs opts: Upstairs UUID: fd0b18fe-644d-4c7c-a791-5a4c8cf83ba3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.312727967Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"Crucible stats registered with UUID: fd0b18fe-644d-4c7c-a791-5a4c8cf83ba3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.312759483Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:10:47.31278041Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.313926026Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.314085456Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.314111412Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.314221219Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.314294059Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206} {"msg":"fd0b18fe-644d-4c7c-a791-5a4c8cf83ba3 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.314326395Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.314347383Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"client":"0","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.314532599Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"client":"1","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.314576164Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"client":"2","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.314614759Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.314638846Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.314664873Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"ds_connection connected from Ok(127.0.0.1:52988)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.315149052Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"ds_connection connected from Ok(127.0.0.1:53315)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.315183837Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"ds_connection connected from Ok(127.0.0.1:60298)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.315207094Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.317904795Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"client":"2","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.318033848Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"client":"1","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.318085352Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"client":"0","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.318980239Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319092525Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319324876Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"downstairs","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319348013Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"downstairs","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.31936919Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"downstairs","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319388088Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"downstairs","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319406345Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"downstairs","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319426153Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"downstairs","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.31944403Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"downstairs","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319462138Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"downstairs","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319481656Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"downstairs","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319499094Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"downstairs","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319515981Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"downstairs","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319531699Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"downstairs","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319552327Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"":"downstairs","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319569235Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319585133Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"client":"0","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319601951Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"client":"1","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319618499Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"client":"2","":"downstairs","session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319635356Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319651484Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"fd0b18fe-644d-4c7c-a791-5a4c8cf83ba3 is now active with session: 8a365dcc-cd88-400e-ba8d-c6dda593006d","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319669272Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.31968437Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206,"session_id":"8a365dcc-cd88-400e-ba8d-c6dda593006d"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.319700448Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206} {"msg":"Activated sub_volume fd0b18fe-644d-4c7c-a791-5a4c8cf83ba3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:47.320152451Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Fill test Read and Verify all blocks (0..600 range:false) Wrote out file "/tmp/test_live_repair/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Read 256 Done Done Done false 1005 Acked Read 256 Done Done Done false 1006 Acked Read 88 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1006 1005 1004 1003 1002 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1007 1006 1005 1004 1003 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:10:51.403866902Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1206} Begin loop 1 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.437341835Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.483518757Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.484230037Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.484280661Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.484297959Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.484320196Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208} {"msg":"Crucible 40e319b0-72a5-463c-b001-4520d020e070 has session id: a6df1b1a-b13e-4584-9bae-620235f47ae2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.484779148Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"Upstairs opts: Upstairs UUID: 40e319b0-72a5-463c-b001-4520d020e070, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.484833731Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"Crucible stats registered with UUID: 40e319b0-72a5-463c-b001-4520d020e070","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.484864288Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:10:51.484892564Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485085909Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485124635Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485144502Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485331499Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"task":"crutest"} {"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485356046Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208} {"msg":"40e319b0-72a5-463c-b001-4520d020e070 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485430366Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485451203Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"client":"0","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485477Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"client":"1","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485516625Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"client":"2","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485552231Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485586817Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485620772Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"ds_connection connected from Ok(127.0.0.1:44120)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485653558Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"ds_connection connected from Ok(127.0.0.1:44750)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.485685054Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"ds_connection connected from Ok(127.0.0.1:48858)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.48571697Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.486888773Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"client":"0","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.486994169Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"client":"1","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.487472379Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.487587294Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.48762181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"client":"2","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488159542Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"downstairs","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488183769Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"downstairs","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488204347Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"downstairs","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488222754Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"downstairs","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488240802Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"downstairs","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.48826136Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"downstairs","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488279507Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"downstairs","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488297505Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"downstairs","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488317173Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"downstairs","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.48833338Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"downstairs","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488350038Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"downstairs","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488371726Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"downstairs","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488387913Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"":"downstairs","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488408661Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488424519Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"client":"0","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488441467Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"client":"1","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488457965Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"client":"2","":"downstairs","session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488474763Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.4884939Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"40e319b0-72a5-463c-b001-4520d020e070 is now active with session: a6df1b1a-b13e-4584-9bae-620235f47ae2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488512218Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488527436Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208,"session_id":"a6df1b1a-b13e-4584-9bae-620235f47ae2"} {"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488543374Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208} {"msg":"Activated sub_volume 40e319b0-72a5-463c-b001-4520d020e070","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:51.488557582Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 334 len 28672 data: 2 2 2 2 2 2 2 03/30 Read block 290 len 12288 04/30 Read block 357 len 32768 05/30 Read block 526 len 4096 06/30 Read block 426 len 40960 07/30 Write block 119 len 40960 data: 2 2 2 2 2 2 2 2 2 2 08/30 Write block 190 len 20480 data: 2 2 2 2 2 09/30 Write block 44 len 16384 data: 2 2 2 2 10/30 Read block 544 len 8192 11/30 Write block 267 len 24576 data: 2 2 2 2 2 2 12/30 Read block 450 len 28672 13/30 Write block 129 len 8192 data: 2 2 14/30 Read block 582 len 4096 15/30 Write block 293 len 12288 data: 2 2 2 16/30 Read block 427 len 40960 17/30 Read block 179 len 12288 18/30 Write block 409 len 40960 data: 2 2 2 2 2 2 2 2 2 2 19/30 Read block 325 len 32768 20/30 Read block 492 len 4096 21/30 Flush 22/30 Write block 582 len 32768 data: 2 2 2 2 2 2 2 2 23/30 Read block 301 len 36864 24/30 Flush 25/30 Write block 313 len 8192 data: 2 2 26/30 Read block 153 len 16384 27/30 Write block 283 len 20480 data: 2 2 2 2 2 28/30 Read block 237 len 20480 29/30 Read block 76 len 36864 30/30 Write block 588 len 36864 data: 3 3 2 2 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 7 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Read 8 Done Done Sent false 1004 Acked Read 1 Done Done Sent false 1005 Acked Read 10 Done Done Sent false 1006 Acked Write 10 Done Done Sent false 1007 Acked Write 5 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Write 6 Done Done Sent false 1011 Acked Read 7 Done Done Sent false 1012 Acked Write 2 Done Done Sent false 1013 Acked Read 1 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Read 10 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Read 8 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 8 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 2 Done Done Sent false 1025 Acked Read 4 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Read 5 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 27 29 Done 28 28 2 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:10:51.508350589Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1208} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 1 1 1 1 1 F F F 2 040-059 2 2 1 3 3 1 F F F <--- 3 060-079 1 1 1 1 1 1 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 2 2 1 3 3 1 F F F <--- 6 120-139 2 2 1 3 3 1 F F F <--- 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 2 2 1 3 3 1 F F F <--- 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 2 2 1 3 3 1 F F F <--- 14 280-299 2 2 1 3 3 1 T T F <--- 15 300-319 1 1 1 1 1 1 T T F <--- 16 320-339 2 2 1 3 3 1 F F T <--- 17 340-359 2 2 1 3 3 1 F F T <--- 18 360-379 1 1 1 1 1 1 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 2 2 1 3 3 1 F F F <--- 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 2 2 1 4 4 1 T T F <--- Max gen: 2, Max flush: 4 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 1 Verifying data now No change after verify Loop: 1 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 1 1 1 1 1 F F F 2 040-059 2 2 2 3 3 3 F F F 3 060-079 1 1 1 1 1 1 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 2 2 2 3 3 3 F F F 6 120-139 2 2 2 3 3 3 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 2 2 2 3 3 3 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 1 1 1 1 1 1 F F F 13 260-279 2 2 2 3 3 3 F F F 14 280-299 3 3 3 5 5 5 F F F 15 300-319 3 3 3 5 5 5 F F F 16 320-339 2 2 2 3 3 3 F F F 17 340-359 2 2 2 3 3 3 F F F 18 360-379 1 1 1 1 1 1 F F F 19 380-399 1 1 1 1 1 1 F F F 20 400-419 2 2 2 3 3 3 F F F 21 420-439 1 1 1 1 1 1 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 1 1 1 1 1 F F F 29 580-599 3 3 3 5 5 5 F F F Max gen: 3, Max flush: 5 Begin loop 2 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.056761952Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:10:56.102105799Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.10249774Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.133922101Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.134623733Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.134672166Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.134689734Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.134717181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216} {"msg":"Crucible edcedc5c-3903-44ba-941c-cd1f31b1b01b has session id: 7cc751cd-cecc-44fc-b8ea-508b6a00eaee","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.135370048Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"Upstairs opts: Upstairs UUID: edcedc5c-3903-44ba-941c-cd1f31b1b01b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.135414143Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"Crucible stats registered with UUID: edcedc5c-3903-44ba-941c-cd1f31b1b01b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.13543545Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:10:56.135452168Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.135644324Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"io task","client":"0","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.135681699Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"io task","client":"1","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.135700427Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"io task","client":"2","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.135826641Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"task":"crutest"} {"msg":"The guest has requested activation with gen:4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.135849898Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216} {"msg":"edcedc5c-3903-44ba-941c-cd1f31b1b01b active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.13591408Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.135934027Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"client":"0","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.135951325Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"client":"1","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.135968393Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"client":"2","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.13599612Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"io task","client":"2","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.136033705Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"io task","client":"0","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.136066791Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"io task","client":"1","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"ds_connection connected from Ok(127.0.0.1:65233)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.136102126Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"io task","client":"0","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"ds_connection connected from Ok(127.0.0.1:49437)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.136136732Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"io task","client":"2","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"ds_connection connected from Ok(127.0.0.1:39711)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.136171607Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"io task","client":"1","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.137383515Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"client":"2","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.137505459Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"client":"1","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.137552364Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"client":"0","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.137994078Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138106584Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 3, 1, 1, 3, 3, 1, 1, 3, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.13821622Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"downstairs","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"[0]R generation[0..12]: [1, 1, 2, 1, 1, 2, 2, 1, 1, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138241627Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"downstairs","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138262334Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"downstairs","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 3, 1, 1, 3, 3, 1, 1, 3, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138280872Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"downstairs","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"[1]R generation[0..12]: [1, 1, 2, 1, 1, 2, 2, 1, 1, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.13829855Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"downstairs","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138322367Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"downstairs","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 3, 1, 1, 3, 3, 1, 1, 3, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138341094Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"downstairs","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"[2]R generation[0..12]: [1, 1, 2, 1, 1, 2, 2, 1, 1, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138359172Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"downstairs","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138378879Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"downstairs","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"Max found gen is 4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138395057Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"Generation requested: 4 >= found:4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138417355Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"downstairs","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"Next flush: 6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138433222Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"downstairs","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138449151Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"":"downstairs","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138466058Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138481376Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"client":"0","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138498164Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"client":"1","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138514932Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"client":"2","":"downstairs","session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.13853172Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138549338Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"edcedc5c-3903-44ba-941c-cd1f31b1b01b is now active with session: 7cc751cd-cecc-44fc-b8ea-508b6a00eaee","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138567885Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138585313Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216,"session_id":"7cc751cd-cecc-44fc-b8ea-508b6a00eaee"} {"msg":"The guest has finished waiting for activation with:4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.138601441Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216} {"msg":"Activated sub_volume edcedc5c-3903-44ba-941c-cd1f31b1b01b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:10:56.13861521Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 585 len 32768 data: 3 3 3 4 4 3 3 3 02/30 Write block 378 len 24576 data: 2 2 2 2 2 2 03/30 Write block 241 len 40960 data: 2 2 2 2 2 2 2 2 2 2 04/30 Write block 434 len 32768 data: 2 2 2 2 2 2 2 2 05/30 Flush 06/30 Write block 20 len 28672 data: 2 2 2 2 2 2 2 07/30 Read block 458 len 40960 08/30 Write block 439 len 12288 data: 3 3 3 09/30 Write block 408 len 16384 data: 2 3 3 3 10/30 Write block 36 len 12288 data: 2 2 2 11/30 Read block 537 len 32768 12/30 Read block 480 len 4096 13/30 Write block 307 len 20480 data: 2 2 2 2 2 14/30 Read block 92 len 16384 15/30 Write block 65 len 16384 data: 2 2 2 2 16/30 Read block 516 len 12288 17/30 Read block 375 len 8192 18/30 Read block 37 len 16384 19/30 Read block 360 len 40960 20/30 Flush 21/30 Write block 564 len 32768 data: 2 2 2 2 2 2 2 2 22/30 Read block 346 len 40960 23/30 Write block 279 len 4096 data: 2 24/30 Read block 538 len 12288 25/30 Read block 316 len 8192 26/30 Flush 27/30 Write block 120 len 20480 data: 3 3 3 3 3 28/30 Write block 57 len 16384 data: 2 2 2 2 29/30 Write block 457 len 12288 data: 2 2 2 30/30 Write block 517 len 36864 data: 2 2 2 2 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:4 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Write 10 Sent Done Done false 1003 Acked Write 8 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 7 Sent Done Done false 1006 Acked Read 10 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Read 8 Sent Done Done false 1011 Acked Read 1 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Read 4 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Read 10 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Read 10 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Write 4 Sent Done Done false 1028 Acked Write 3 Sent Sent Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 2 1 31 Done 2 28 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:10:56.156762143Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1216} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 1 4 4 1 7 7 F F F <--- 2 040-059 2 2 2 3 3 3 F T T <--- 3 060-079 1 4 4 1 7 7 F T T <--- 4 080-099 1 1 1 1 1 1 F F F 5 100-119 2 2 2 3 3 3 F F F 6 120-139 2 2 2 3 3 3 F T T <--- 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 2 2 2 3 3 3 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 1 4 4 1 6 6 F F F <--- 13 260-279 2 4 4 3 8 8 F F F <--- 14 280-299 3 3 3 5 5 5 F F F 15 300-319 3 4 4 5 7 7 F F F <--- 16 320-339 2 2 2 3 3 3 F F F 17 340-359 2 2 2 3 3 3 F F F 18 360-379 1 4 4 1 6 6 T F F <--- 19 380-399 1 4 4 1 6 6 T F F <--- 20 400-419 2 4 4 3 7 7 F F F <--- 21 420-439 1 4 4 1 7 7 F F F <--- 22 440-459 1 4 4 1 7 7 F T T <--- 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 1 1 1 1 1 1 F T T <--- 26 520-539 1 1 1 1 1 1 F T T <--- 27 540-559 1 1 1 1 1 1 F F F 28 560-579 1 4 4 1 8 8 F F F <--- 29 580-599 3 4 4 5 6 6 T F F <--- Max gen: 4, Max flush: 8 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 2 Verifying data now No change after verify Loop: 2 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 4 4 4 7 7 7 F F F 2 040-059 5 5 5 9 9 9 F F F 3 060-079 5 5 5 9 9 9 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 2 2 2 3 3 3 F F F 6 120-139 5 5 5 9 9 9 F F F 7 140-159 1 1 1 1 1 1 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 2 2 2 3 3 3 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 4 4 4 8 8 8 F F F 14 280-299 3 3 3 5 5 5 F F F 15 300-319 4 4 4 7 7 7 F F F 16 320-339 2 2 2 3 3 3 F F F 17 340-359 2 2 2 3 3 3 F F F 18 360-379 4 4 4 6 6 6 F F F 19 380-399 4 4 4 6 6 6 F F F 20 400-419 4 4 4 7 7 7 F F F 21 420-439 4 4 4 7 7 7 F F F 22 440-459 5 5 5 9 9 9 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 5 5 5 9 9 9 F F F 26 520-539 5 5 5 9 9 9 F F F 27 540-559 1 1 1 1 1 1 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 4 4 4 6 6 6 F F F Max gen: 5, Max flush: 9 Begin loop 3 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.618839995Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.664248854Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.664886284Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.664935388Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.664952895Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.664975103Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224} {"msg":"Crucible d55fd772-0cb1-445c-b6e8-979b6f3ea6af has session id: f72235bb-399a-41f4-a9a1-73f0e2d068b4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.665423626Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"Upstairs opts: Upstairs UUID: d55fd772-0cb1-445c-b6e8-979b6f3ea6af, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.665530673Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"Crucible stats registered with UUID: d55fd772-0cb1-445c-b6e8-979b6f3ea6af","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.665568198Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:00.665594945Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.665782271Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"io task","client":"0","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.665820086Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"io task","client":"2","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.665843463Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"io task","client":"1","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.666007813Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"task":"crutest"} {"msg":"The guest has requested activation with gen:6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.666049887Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224} {"msg":"d55fd772-0cb1-445c-b6e8-979b6f3ea6af active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.666131357Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.666155694Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"client":"0","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.666173462Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"client":"1","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.66619045Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"client":"2","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.666213587Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"io task","client":"2","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.666242593Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"io task","client":"0","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.666274339Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"io task","client":"1","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:45180)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.666364438Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"io task","client":"2","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:49397)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.666401443Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"io task","client":"1","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:33237)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.666431739Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"io task","client":"0","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.667735675Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"client":"0","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.667770911Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"client":"1","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.667792708Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"client":"2","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.66832996Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668358177Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"[0]R flush_numbers[0..12]: [1, 7, 9, 9, 1, 3, 9, 1, 1, 3, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668388962Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"downstairs","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"[0]R generation[0..12]: [1, 4, 5, 5, 1, 2, 5, 1, 1, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.66840816Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"downstairs","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668428308Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"downstairs","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"[1]R flush_numbers[0..12]: [1, 7, 9, 9, 1, 3, 9, 1, 1, 3, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668446745Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"downstairs","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"[1]R generation[0..12]: [1, 4, 5, 5, 1, 2, 5, 1, 1, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668464863Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"downstairs","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.66848453Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"downstairs","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"[2]R flush_numbers[0..12]: [1, 7, 9, 9, 1, 3, 9, 1, 1, 3, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668502608Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"downstairs","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"[2]R generation[0..12]: [1, 4, 5, 5, 1, 2, 5, 1, 1, 2, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668520716Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"downstairs","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668540274Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"downstairs","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"Max found gen is 6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668556381Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"downstairs","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"Generation requested: 6 >= found:6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668573049Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"downstairs","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"Next flush: 10","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668589097Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"downstairs","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668605145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"":"downstairs","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} Wait for a query_work_queue command to finish before sending IO {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668631732Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.66864793Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"client":"0","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668664938Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"client":"1","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668681526Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"client":"2","":"downstairs","session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668698623Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668714861Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"d55fd772-0cb1-445c-b6e8-979b6f3ea6af is now active with session: f72235bb-399a-41f4-a9a1-73f0e2d068b4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668732879Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668748227Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224,"session_id":"f72235bb-399a-41f4-a9a1-73f0e2d068b4"} {"msg":"The guest has finished waiting for activation with:6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668764055Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224} {"msg":"Activated sub_volume d55fd772-0cb1-445c-b6e8-979b6f3ea6af","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:00.668778093Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 478 len 8192 data: 2 2 02/30 Read block 591 len 20480 03/30 Read block 225 len 40960 04/30 Read block 28 len 4096 05/30 Read block 218 len 28672 06/30 Flush 07/30 Flush 08/30 Flush 09/30 Write block 186 len 16384 data: 2 2 2 2 10/30 Read block 422 len 24576 11/30 Write block 38 len 40960 data: 3 2 2 2 2 2 3 3 3 3 12/30 Read block 316 len 20480 13/30 Read block 408 len 12288 14/30 Read block 493 len 28672 15/30 Read block 61 len 28672 16/30 Read block 361 len 28672 17/30 Write block 387 len 8192 data: 2 2 18/30 Read block 472 len 20480 19/30 Read block 329 len 24576 20/30 Read block 154 len 40960 21/30 Write block 36 len 20480 data: 3 3 4 3 3 22/30 Write block 159 len 8192 data: 2 2 23/30 Write block 95 len 24576 data: 2 2 2 2 2 2 24/30 Read block 392 len 28672 25/30 Read block 188 len 24576 26/30 Flush 27/30 Write block 512 len 40960 data: 2 2 2 2 2 3 3 3 3 3 28/30 Read block 425 len 20480 29/30 Write block 542 len 36864 data: 2 2 2 2 2 2 2 2 2 30/30 Write block 25 len 4096 data: 3 ---------------------------------------------------------------- Crucible gen:6 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Sent false 1001 Acked Read 5 Done Done Sent false 1002 Acked Read 10 Done Done Sent false 1003 Acked Read 1 Done Done Sent false 1004 Acked Read 7 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Write 10 Done Done Sent false 1011 Acked Read 5 Done Done Sent false 1012 Acked Read 3 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Write 2 Done Done Sent false 1017 Acked Read 5 Done Done Sent false 1018 Acked Read 6 Done Done Sent false 1019 Acked Read 10 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Write 6 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Read 6 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Read 5 Done Done Sent false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 30 34 Done 28 28 0 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:00.687621219Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1224} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 6 6 4 13 13 7 T T F <--- 2 040-059 6 6 5 13 13 9 F F F <--- 3 060-079 5 5 5 9 9 9 F F F 4 080-099 6 6 1 13 13 1 F F F <--- 5 100-119 6 6 2 13 13 3 F F F <--- 6 120-139 5 5 5 9 9 9 F F F 7 140-159 6 6 1 13 13 1 F F F <--- 8 160-179 6 6 1 13 13 1 F F F <--- 9 180-199 6 6 2 13 13 3 F F F <--- 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 4 4 4 8 8 8 F F F 14 280-299 3 3 3 5 5 5 F F F 15 300-319 4 4 4 7 7 7 F F F 16 320-339 2 2 2 3 3 3 F F F 17 340-359 2 2 2 3 3 3 F F F 18 360-379 4 4 4 6 6 6 F F F 19 380-399 6 6 4 13 13 6 F F F <--- 20 400-419 4 4 4 7 7 7 F F F 21 420-439 4 4 4 7 7 7 F F F 22 440-459 5 5 5 9 9 9 F F F 23 460-479 6 6 1 10 10 1 F F F <--- 24 480-499 1 1 1 1 1 1 F F F 25 500-519 5 5 5 9 9 9 T T F <--- 26 520-539 5 5 5 9 9 9 T T F <--- 27 540-559 1 1 1 1 1 1 T T F <--- 28 560-579 4 4 4 8 8 8 F F F 29 580-599 4 4 4 6 6 6 F F F Max gen: 6, Max flush: 13 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 3 Verifying data now No change after verify Loop: 3 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 7 7 7 14 14 14 F F F 2 040-059 6 6 6 13 13 13 F F F 3 060-079 5 5 5 9 9 9 F F F 4 080-099 6 6 6 13 13 13 F F F 5 100-119 6 6 6 13 13 13 F F F 6 120-139 5 5 5 9 9 9 F F F 7 140-159 6 6 6 13 13 13 F F F 8 160-179 6 6 6 13 13 13 F F F 9 180-199 6 6 6 13 13 13 F F F 10 200-219 1 1 1 1 1 1 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 4 4 4 6 6 6 F F F 13 260-279 4 4 4 8 8 8 F F F 14 280-299 3 3 3 5 5 5 F F F 15 300-319 4 4 4 7 7 7 F F F 16 320-339 2 2 2 3 3 3 F F F 17 340-359 2 2 2 3 3 3 F F F 18 360-379 4 4 4 6 6 6 F F F 19 380-399 6 6 6 13 13 13 F F F 20 400-419 4 4 4 7 7 7 F F F 21 420-439 4 4 4 7 7 7 F F F 22 440-459 5 5 5 9 9 9 F F F 23 460-479 6 6 6 10 10 10 F F F 24 480-499 1 1 1 1 1 1 F F F 25 500-519 7 7 7 14 14 14 F F F 26 520-539 7 7 7 14 14 14 F F F 27 540-559 7 7 7 14 14 14 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 4 4 4 6 6 6 F F F Max gen: 7, Max flush: 14 Begin loop 4 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.080233482Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.125282736Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.125899778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.125958411Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.125979778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.126017424Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232} {"msg":"Crucible de3a718b-1d5e-4c1a-90f0-18f77791c179 has session id: 732a6b0d-585e-4ede-a820-5d6169c8c935","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.12644212Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"Upstairs opts: Upstairs UUID: de3a718b-1d5e-4c1a-90f0-18f77791c179, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.126514251Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"Crucible stats registered with UUID: de3a718b-1d5e-4c1a-90f0-18f77791c179","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.126554196Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:05.126587742Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.126738513Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"io task","client":"0","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.126783677Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"io task","client":"1","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.126812763Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.127007319Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"task":"crutest"} {"msg":"The guest has requested activation with gen:8","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.127036255Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232} {"msg":"de3a718b-1d5e-4c1a-90f0-18f77791c179 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.127090568Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.127120845Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"client":"0","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.127140782Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"client":"1","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.127159729Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"client":"2","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.127187216Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.127211693Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"io task","client":"0","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.127229851Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"io task","client":"1","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"ds_connection connected from Ok(127.0.0.1:48115)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.127252298Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"io task","client":"2","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"ds_connection connected from Ok(127.0.0.1:53508)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.127330958Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"io task","client":"0","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"ds_connection connected from Ok(127.0.0.1:52245)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.127440964Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"io task","client":"1","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.128441708Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"client":"1","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.128489242Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"client":"0","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.129080148Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.129155098Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.129711958Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"client":"2","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"[0]R flush_numbers[0..12]: [1, 14, 13, 9, 13, 13, 9, 13, 13, 13, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130196567Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"downstairs","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"[0]R generation[0..12]: [1, 7, 6, 5, 6, 6, 5, 6, 6, 6, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130224363Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"downstairs","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130245441Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"downstairs","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"[1]R flush_numbers[0..12]: [1, 14, 13, 9, 13, 13, 9, 13, 13, 13, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130264178Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"downstairs","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"[1]R generation[0..12]: [1, 7, 6, 5, 6, 6, 5, 6, 6, 6, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130282376Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"downstairs","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130302244Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"downstairs","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"[2]R flush_numbers[0..12]: [1, 14, 13, 9, 13, 13, 9, 13, 13, 13, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130324351Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"downstairs","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"[2]R generation[0..12]: [1, 7, 6, 5, 6, 6, 5, 6, 6, 6, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130343099Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"downstairs","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130362906Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"downstairs","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"Max found gen is 8","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130379384Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"downstairs","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"Generation requested: 8 >= found:8","v":0,"name":"crucible",Wait for a query_work_queue command to finish before sending IO "level":30,"time":"2025-11-06T20:11:05.130403661Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"downstairs","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"Next flush: 15","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130420089Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"downstairs","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130436447Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"":"downstairs","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130453365Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130468623Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"client":"0","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.13048553Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"client":"1","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130502579Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"client":"2","":"downstairs","session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130519936Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130536064Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"de3a718b-1d5e-4c1a-90f0-18f77791c179 is now active with session: 732a6b0d-585e-4ede-a820-5d6169c8c935","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130554222Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.13056962Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232,"session_id":"732a6b0d-585e-4ede-a820-5d6169c8c935"} {Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "msg":"The guest has finished waiting for activation with:8","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130591977Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232} {"msg":"Activated sub_volume de3a718b-1d5e-4c1a-90f0-18f77791c179","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:05.130606135Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 584 len 36864 data: 3 4 4 4 5 5 4 4 4 02/30 Write block 541 len 8192 data: 2 3 03/30 Write block 440 len 12288 data: 4 4 2 04/30 Write block 383 len 16384 data: 3 2 2 2 05/30 Flush 06/30 Write block 586 len 20480 data: 5 5 6 6 5 07/30 Read block 594 len 24576 08/30 Read block 58 len 32768 09/30 Write block 250 len 36864 data: 3 2 2 2 2 2 2 2 2 10/30 Write block 498 len 24576 data: 2 2 2 2 2 2 11/30 Write block 161 len 8192 data: 2 2 12/30 Write block 397 len 4096 data: 2 13/30 Read block 297 len 36864 14/30 Write block 206 len 20480 data: 2 2 2 2 2 15/30 Read block 519 len 28672 16/30 Read block 252 len 24576 17/30 Read block 434 len 12288 18/30 Read block 44 len 8192 19/30 Read block 401 len 40960 20/30 Write block 89 len 4096 data: 2 21/30 Flush 22/30 Write block 181 len 28672 data: 2 2 2 2 2 3 3 23/30 Write block 451 len 36864 data: 2 2 2 2 2 2 3 3 3 24/30 Read block 540 len 36864 25/30 Write block 127 len 4096 data: 3 26/30 Read block 402 len 36864 27/30 Read block 391 len 16384 28/30 Write block 323 len 32768 data: 2 2 2 2 2 2 2 2 29/30 Read block 245 len 4096 30/30 Read block 495 len 24576 ---------------------------------------------------------------- Crucible gen:8 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Write 3 Done Done Sent false 1003 Acked Write 4 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Read 6 Done Done Sent false 1007 Acked Read 8 Done Done Sent false 1008 Acked Write 9 Done Done Sent false 1009 Acked Write 6 Done Done Sent false 1010 Acked Write 2 Done Done Sent false 1011 Acked Write 1 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Write 5 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Read 6 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Write 1 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 7 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Read 9 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Read 4 Done Done Sent false 1027 Acked Write 8 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Read 6 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 28 28 Done 30 30 2 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:05.149414586Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1232} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 7 7 7 14 14 14 F F F 2 040-059 6 6 6 13 13 13 F F F 3 060-079 5 5 5 9 9 9 F F F 4 080-099 8 8 6 16 16 13 F F F <--- 5 100-119 6 6 6 13 13 13 F F F 6 120-139 5 5 5 9 9 9 T T F <--- 7 140-159 6 6 6 13 13 13 F F F 8 160-179 8 8 6 16 16 13 F F F <--- 9 180-199 6 6 6 13 13 13 T T F <--- 10 200-219 8 8 1 16 16 1 F F F <--- 11 220-239 1 1 1 1 1 1 F F F 12 240-259 8 8 4 16 16 6 F F F <--- 13 260-279 4 4 4 8 8 8 F F F 14 280-299 3 3 3 5 5 5 F F F 15 300-319 4 4 4 7 7 7 F F F 16 320-339 2 2 2 3 3 3 T T F <--- 17 340-359 2 2 2 3 3 3 F F F 18 360-379 4 4 4 6 6 6 F F F 19 380-399 8 8 6 16 16 13 F F F <--- 20 400-419 4 4 4 7 7 7 F F F 21 420-439 4 4 4 7 7 7 F F F 22 440-459 8 8 5 15 15 9 T T F <--- 23 460-479 6 6 6 10 10 10 F F F 24 480-499 8 8 1 16 16 1 F F F <--- 25 500-519 8 8 7 16 16 14 F F F <--- 26 520-539 7 7 7 14 14 14 F F F 27 540-559 8 8 7 15 15 14 F F T <--- 28 560-579 4 4 4 8 8 8 F F F 29 580-599 8 8 4 16 16 6 F F T <--- Max gen: 8, Max flush: 16 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 4 Verifying data now No change after verify Loop: 4 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 7 7 7 14 14 14 F F F 2 040-059 6 6 6 13 13 13 F F F 3 060-079 5 5 5 9 9 9 F F F 4 080-099 8 8 8 16 16 16 F F F 5 100-119 6 6 6 13 13 13 F F F 6 120-139 9 9 9 17 17 17 F F F 7 140-159 6 6 6 13 13 13 F F F 8 160-179 8 8 8 16 16 16 F F F 9 180-199 9 9 9 17 17 17 F F F 10 200-219 8 8 8 16 16 16 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 8 8 8 16 16 16 F F F 13 260-279 4 4 4 8 8 8 F F F 14 280-299 3 3 3 5 5 5 F F F 15 300-319 4 4 4 7 7 7 F F F 16 320-339 9 9 9 17 17 17 F F F 17 340-359 2 2 2 3 3 3 F F F 18 360-379 4 4 4 6 6 6 F F F 19 380-399 8 8 8 16 16 16 F F F 20 400-419 4 4 4 7 7 7 F F F 21 420-439 4 4 4 7 7 7 F F F 22 440-459 9 9 9 17 17 17 F F F 23 460-479 6 6 6 10 10 10 F F F 24 480-499 8 8 8 16 16 16 F F F 25 500-519 8 8 8 16 16 16 F F F 26 520-539 7 7 7 14 14 14 F F F 27 540-559 8 8 8 15 15 15 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 8 8 8 16 16 16 F F F Max gen: 9, Max flush: 17 Begin loop 5 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.55090972Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.597694356Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.598402787Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.59845106Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.598467318Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.598489406Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240} {"msg":"Crucible cbc5c101-6e10-4f5f-b297-0afef1f3a322 has session id: 280e585f-e886-4b1e-8679-21909f7b210f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.599037537Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"Upstairs opts: Upstairs UUID: cbc5c101-6e10-4f5f-b297-0afef1f3a322, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.599079851Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"Crucible stats registered with UUID: cbc5c101-6e10-4f5f-b297-0afef1f3a322","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.599110037Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:09.599138674Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.599375614Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"io task","client":"1","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.599420938Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"io task","client":"2","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.599441976Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"io task","client":"0","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.59964392Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"task":"crutest"} {"msg":"The guest has requested activation with gen:10","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.599681076Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240} {"msg":"cbc5c101-6e10-4f5f-b297-0afef1f3a322 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.599788852Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.599834856Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"client":"0","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.59995921Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"client":"1","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.599997766Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"client":"2","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.600030032Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"io task","client":"2","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.600061058Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"io task","client":"0","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.600092824Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"io task","client":"1","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"ds_connection connected from Ok(127.0.0.1:62888)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.600128219Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"io task","client":"1","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"ds_connection connected from Ok(127.0.0.1:49115)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.600239725Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"io task","client":"0","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"ds_connection connected from Ok(127.0.0.1:60096)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.60027993Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"io task","client":"2","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.60123265Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"client":"1","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.601340816Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"client":"0","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.601407128Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"client":"2","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.601809137Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.60187048Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"[0]R flush_numbers[0..12]: [1, 14, 13, 9, 16, 13, 17, 13, 16, 17, 16, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.601970717Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"downstairs","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"[0]R generation[0..12]: [1, 7, 6, 5, 8, 6, 9, 6, 8, 9, 8, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.601994514Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"downstairs","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602014822Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"downstairs","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"[1]R flush_numbers[0..12]: [1, 14, 13, 9, 16, 13, 17, 13, 16, 17, 16, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602033379Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"downstairs","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"[1]R generation[0..12]: [1, 7, 6, 5, 8, 6, 9, 6, 8, 9, 8, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602051327Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"downstairs","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602070804Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"downstairs","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"[2]R flush_numbers[0..12]: [1, 14, 13, 9, 16, 13, 17, 13, 16, 17, 16, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602088972Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"downstairs","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"[2]R generation[0..12]: [1, 7, 6, 5, 8, 6, 9, 6, 8, 9, 8, 1]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.60210692Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"downstairs","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602126628Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"downstairs","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"Max found gen is 10","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602143815Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid"Wait for a query_work_queue command to finish before sending IO :1240,"":"downstairs","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"Generation requested: 10 >= found:10","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602167802Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"downstairs","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"Next flush: 18","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.60218404Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"downstairs","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602200158Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"":"downstairs","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602216976Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602235194Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"client":"0","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602252112Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"client":"1","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602268799Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"client":"2","":"downstairs","session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602285967Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602301975Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"cbc5c101-6e10-4f5f-b297-0afef1f3a322 is now active with session: 280e585f-e886-4b1e-8679-21909f7b210f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602319803Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602334851Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240,"session_id":"280e585f-e886-4b1e-8679-21909f7b210f"} {"msg":"The guest has finished waiting for activation with:10","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602350799Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240} {"msg":"Activated sub_volume cbc5c101-6e10-4f5f-b297-0afef1f3a322","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:09.602364678Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 394 len 32768 data: 2 2 2 3 2 2 2 2 02/30 Read block 582 len 28672 03/30 Read block 35 len 28672 04/30 Read block 181 len 12288 05/30 Read block 199 len 24576 06/30 Read block 283 len 32768 07/30 Write block 223 len 4096 data: 2 08/30 Read block 287 len 32768 09/30 Read block 392 len 36864 10/30 Flush 11/30 Write block 209 len 36864 data: 3 3 2 2 2 2 2 2 2 12/30 Read block 92 len 4096 13/30 Read block 269 len 4096 14/30 Write block 207 len 4096 data: 3 15/30 Write block 102 len 12288 data: 2 2 2 16/30 Flush 17/30 Write block 132 len 20480 data: 2 2 2 2 2 18/30 Write block 179 len 20480 data: 2 2 3 3 3 19/30 Read block 544 len 36864 20/30 Read block 181 len 28672 21/30 Write block 97 len 4096 data: 3 22/30 Read block 436 len 24576 23/30 Write block 332 len 8192 data: 2 2 24/30 Write block 307 len 20480 data: 3 3 3 3 3 25/30 Read block 9 len 12288 26/30 Read block 390 len 12288 27/30 Read block 572 len 32768 28/30 Write block 79 len 40960 data: 2 2 2 2 2 2 2 2 2 2 29/30 Write block 34 len 36864 data: 2 2 4 4 5 4 4 3 3 30/30 Write block 424 len 32768 data: 2 2 2 2 2 2 2 2 ---------------------------------------------------------------- Crucible gen:10 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 7 Done Done Sent false 1002 Acked Read 7 Done Done Sent false 1003 Acked Read 3 Done Done Sent false 1004 Acked Read 6 Done Done Sent false 1005 Acked Read 8 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Read 8 Done Done Sent false 1008 Acked Read 9 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 9 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Write 1 Done Done Sent false 1021 Acked Read 6 Done Done Sent false 1022 Acked Write 2 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Read 3 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Read 8 Done Done Sent false 1027 Acked Write 10 Done Done Sent false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 29 33 Done 28 28 1 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) Some(JobId(1015)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:09.621044094Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1240} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 7 7 7 14 14 14 T T F <--- 2 040-059 6 6 6 13 13 13 T T F <--- 3 060-079 5 5 5 9 9 9 T T F <--- 4 080-099 8 8 8 16 16 16 T T F <--- 5 100-119 10 10 6 19 19 13 F F F <--- 6 120-139 9 9 9 17 17 17 T T F <--- 7 140-159 6 6 6 13 13 13 F F F 8 160-179 8 8 8 16 16 16 T T F <--- 9 180-199 9 9 9 17 17 17 T T F <--- 10 200-219 10 10 8 19 19 16 F F F <--- 11 220-239 10 10 1 18 18 1 F F F <--- 12 240-259 8 8 8 16 16 16 F F F 13 260-279 4 4 4 8 8 8 F F F 14 280-299 3 3 3 5 5 5 F F F 15 300-319 4 4 4 7 7 7 T T F <--- 16 320-339 9 9 9 17 17 17 T T F <--- 17 340-359 2 2 2 3 3 3 F F F 18 360-379 4 4 4 6 6 6 F F F 19 380-399 10 10 8 18 18 16 F F T <--- 20 400-419 10 10 4 18 18 7 F F T <--- 21 420-439 4 4 4 7 7 7 T T F <--- 22 440-459 9 9 9 17 17 17 F F F 23 460-479 6 6 6 10 10 10 F F F 24 480-499 8 8 8 16 16 16 F F F 25 500-519 8 8 8 16 16 16 F F F 26 520-539 7 7 7 14 14 14 F F F 27 540-559 8 8 8 15 15 15 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 8 8 8 16 16 16 F F F Max gen: 10, Max flush: 19 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 5 Verifying data now No change after verify Loop: 5 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 11 11 11 20 20 20 F F F 2 040-059 11 11 11 20 20 20 F F F 3 060-079 11 11 11 20 20 20 F F F 4 080-099 11 11 11 20 20 20 F F F 5 100-119 10 10 10 19 19 19 F F F 6 120-139 11 11 11 20 20 20 F F F 7 140-159 6 6 6 13 13 13 F F F 8 160-179 11 11 11 20 20 20 F F F 9 180-199 11 11 11 20 20 20 F F F 10 200-219 10 10 10 19 19 19 F F F 11 220-239 10 10 10 18 18 18 F F F 12 240-259 8 8 8 16 16 16 F F F 13 260-279 4 4 4 8 8 8 F F F 14 280-299 3 3 3 5 5 5 F F F 15 300-319 11 11 11 20 20 20 F F F 16 320-339 11 11 11 20 20 20 F F F 17 340-359 2 2 2 3 3 3 F F F 18 360-379 4 4 4 6 6 6 F F F 19 380-399 10 10 10 18 18 18 F F F 20 400-419 10 10 10 18 18 18 F F F 21 420-439 11 11 11 20 20 20 F F F 22 440-459 9 9 9 17 17 17 F F F 23 460-479 6 6 6 10 10 10 F F F 24 480-499 8 8 8 16 16 16 F F F 25 500-519 8 8 8 16 16 16 F F F 26 520-539 7 7 7 14 14 14 F F F 27 540-559 8 8 8 15 15 15 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 8 8 8 16 16 16 F F F Max gen: 11, Max flush: 20 Begin loop 6 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.034979761Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.080717519Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.081354539Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.081404662Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.08142138Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.081449047Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248} {"msg":"Crucible 1e68ac18-7605-462e-88b5-9ff26fd92858 has session id: d570eee2-296e-4ced-8334-5f2420de836a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.081986779Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Upstairs opts: Upstairs UUID: 1e68ac18-7605-462e-88b5-9ff26fd92858, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.082047031Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Crucible stats registered with UUID: 1e68ac18-7605-462e-88b5-9ff26fd92858","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.082076657Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:14.082094415Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.082356772Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"io task","client":"2","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.082397317Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"io task","client":"1","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.082430653Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"io task","client":"0","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.082626118Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"task":"crutest"} {"msg":"The guest has requested activation with gen:12","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.082653825Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248} {"msg":"1e68ac18-7605-462e-88b5-9ff26fd92858 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.082719456Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.082739694Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"client":"0","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.082758092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"client":"1","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.082786458Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"client":"2","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.082817084Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"io task","client":"2","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.08285003Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"io task","client":"0","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.082881236Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"io task","client":"1","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"ds_connection connected from Ok(127.0.0.1:64894)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.08308985Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"io task","client":"2","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"ds_connection connected from Ok(127.0.0.1:38445)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.083128255Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"io task","client":"0","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"ds_connection connected from Ok(127.0.0.1:40549)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.083149812Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"io task","client":"1","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.084194551Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"client":"0","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.084314886Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"client":"1","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.084348931Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"client":"2","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.084744671Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.084821282Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"[0]R flush_numbers[0..12]: [1, 20, 20, 20, 20, 19, 20, 13, 20, 20, 19, 18]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.084921409Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"downstairs","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"[0]R generation[0..12]: [1, 11, 11, 11, 11, 10, 11, 6, 11, 11, 10, 10]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.084946116Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"downstairs","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.084966464Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"downstairs","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"[1]R flush_numbers[0..12]: [1, 20, 20, 20, 20, 19, 20, 13, 20, 20, 19, 18]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.084985211Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"downstairs","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"[1]R generation[0..12]: [1, 11, 11, 11, 11, 10, 11, 6, 11, 11, 10, 10]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085003809Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"downstairs","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085023576Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"downstairs","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"[2]R flush_numbers[0..12]: [1, 20, 20, 20, 20, 19, 20, 13, 20, 20, 19, 18]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085042154Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"downstairs","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"[2]R generation[0..12]: [1, 11, 11, 11, 11, 10, 11, 6, 11, 11, 10, 10]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085063681Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"downstairs","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085083679Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"downstairs","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Max found gen is 12","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085099887Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"downstairs","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Generation requested: 12 >= found:12","v":0,"Wait for a query_work_queue command to finish before sending IO name":"crucible","level":30,"time":"2025-11-06T20:11:14.085123824Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"downstairs","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Next flush: 21","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085139982Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"downstairs","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.08515604Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"":"downstairs","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085173267Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085188705Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"client":"0","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085205873Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"client":"1","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085222781Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"client":"2","":"downstairs","session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085240089Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085256187Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"1e68ac18-7605-462e-88b5-9ff26fd92858 is now active with session: d570eee2-296e-4ced-8334-5f2420de836a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085274195Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085289253Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248,"session_id":"d570eee2-296e-4ced-8334-5f2420de836a"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 The guest has finished waiting for activation with:12","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.08530771Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248} {"msg":"Activated sub_volume 1e68ac18-7605-462e-88b5-9ff26fd92858","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:14.085323848Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 453 len 20480 data: 3 3 3 3 4 02/30 Flush 03/30 Write block 377 len 4096 data: 2 04/30 Write block 467 len 40960 data: 2 2 2 2 2 2 2 2 2 2 05/30 Read block 308 len 16384 06/30 Read block 203 len 40960 07/30 Write block 39 len 40960 data: 5 5 4 4 3 4 4 4 4 2 08/30 Write block 258 len 4096 data: 3 09/30 Write block 540 len 36864 data: 2 3 4 3 3 3 3 3 3 10/30 Write block 588 len 12288 data: 7 7 6 11/30 Read block 153 len 36864 12/30 Flush 13/30 Write block 165 len 32768 data: 2 2 2 2 2 2 2 2 14/30 Read block 521 len 8192 15/30 Read block 371 len 16384 16/30 Write block 434 len 12288 data: 3 3 3 17/30 Read block 398 len 36864 18/30 Write block 339 len 12288 data: 3 3 2 19/30 Read block 483 len 20480 20/30 Read block 378 len 12288 21/30 Write block 312 len 12288 data: 2 3 3 22/30 Read block 304 len 36864 23/30 Write block 164 len 24576 data: 2 3 3 3 3 3 24/30 Read block 272 len 16384 25/30 Flush 26/30 Write block 427 len 36864 data: 3 3 3 3 3 2 2 4 4 27/30 Write block 128 len 16384 data: 3 3 3 2 28/30 Write block 436 len 40960 data: 4 3 3 4 5 5 3 2 2 2 29/30 Read block 185 len 16384 30/30 Read block 424 len 28672 ---------------------------------------------------------------- Crucible gen:12 GIO:true work queues: Upstairs:0 downstairs:18 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1012 Acked Write 8 Done Sent Done false 1013 Acked Read 2 Done Sent Done false 1014 Acked Read 4 Done Sent Done false 1015 Acked Write 3 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Write 3 Done Sent Done false 1018 Acked Read 5 Done Sent Done false 1019 Acked Read 3 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Read 7 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 18 0 18 Done 18 0 18 36 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1011)) Some(JobId(1024)) Downstairs last five completed: 1011 1010 1009 1008 1007 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:14.10475984Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1248} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 12 12 12 22 22 22 F F F 2 040-059 12 12 12 22 22 22 F F F 3 060-079 11 11 11 20 20 20 F F F 4 080-099 11 11 11 20 20 20 F F F 5 100-119 10 10 10 19 19 19 F F F 6 120-139 11 11 11 20 20 20 T F T <--- 7 140-159 6 6 6 13 13 13 F F F 8 160-179 12 11 12 23 20 23 F F F <--- 9 180-199 11 11 11 20 20 20 F F F 10 200-219 10 10 10 19 19 19 F F F 11 220-239 10 10 10 18 18 18 F F F 12 240-259 12 12 12 22 22 22 F F F 13 260-279 4 4 4 8 8 8 F F F 14 280-299 3 3 3 5 5 5 F F F 15 300-319 12 11 12 23 20 23 F F F <--- 16 320-339 12 11 12 23 20 23 F F F <--- 17 340-359 12 2 12 23 3 23 F F F <--- 18 360-379 12 12 12 22 22 22 F F F 19 380-399 10 10 10 18 18 18 F F F 20 400-419 10 10 10 18 18 18 F F F 21 420-439 12 11 12 23 20 23 T F T <--- 22 440-459 12 12 12 21 21 21 T F T <--- 23 460-479 12 12 12 22 22 22 F F F 24 480-499 8 8 8 16 16 16 F F F 25 500-519 8 8 8 16 16 16 F F F 26 520-539 7 7 7 14 14 14 F F F 27 540-559 12 12 12 22 22 22 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 12 12 12 22 22 22 F F F Max gen: 12, Max flush: 23 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 6 Verifying data now No change after verify Loop: 6 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 F F F 1 020-039 12 12 12 22 22 22 F F F 2 040-059 12 12 12 22 22 22 F F F 3 060-079 11 11 11 20 20 20 F F F 4 080-099 11 11 11 20 20 20 F F F 5 100-119 10 10 10 19 19 19 F F F 6 120-139 13 13 13 24 24 24 F F F 7 140-159 6 6 6 13 13 13 F F F 8 160-179 12 12 12 23 23 23 F F F 9 180-199 11 11 11 20 20 20 F F F 10 200-219 10 10 10 19 19 19 F F F 11 220-239 10 10 10 18 18 18 F F F 12 240-259 12 12 12 22 22 22 F F F 13 260-279 4 4 4 8 8 8 F F F 14 280-299 3 3 3 5 5 5 F F F 15 300-319 12 12 12 23 23 23 F F F 16 320-339 12 12 12 23 23 23 F F F 17 340-359 12 12 12 23 23 23 F F F 18 360-379 12 12 12 22 22 22 F F F 19 380-399 10 10 10 18 18 18 F F F 20 400-419 10 10 10 18 18 18 F F F 21 420-439 13 13 13 24 24 24 F F F 22 440-459 13 13 13 24 24 24 F F F 23 460-479 12 12 12 22 22 22 F F F 24 480-499 8 8 8 16 16 16 F F F 25 500-519 8 8 8 16 16 16 F F F 26 520-539 7 7 7 14 14 14 F F F 27 540-559 12 12 12 22 22 22 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 12 12 12 22 22 22 F F F Max gen: 13, Max flush: 24 Begin loop 7 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.479820375Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.525386704Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.526010415Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.526075507Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.526097435Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.526121401Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256} {"msg":"Crucible fc345908-83dc-43e0-ae8c-d2c8c0b00db2 has session id: 62266807-fb88-4de1-b939-ad08d6ff89ec","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.52660618Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"Upstairs opts: Upstairs UUID: fc345908-83dc-43e0-ae8c-d2c8c0b00db2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.526671052Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"Crucible stats registered with UUID: fc345908-83dc-43e0-ae8c-d2c8c0b00db2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.526702608Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:18.526721496Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.526905133Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"io task","client":"0","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.526956466Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"io task","client":"2","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.526988522Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"io task","client":"1","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.52732374Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"task":"crutest"} {"msg":"The guest has requested activation with gen:14","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.527352177Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256} {"msg":"fc345908-83dc-43e0-ae8c-d2c8c0b00db2 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.527418598Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.527439195Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"client":"0","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.527460203Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"client":"1","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.527477741Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"client":"2","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.527501448Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"io task","client":"2","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.527532214Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"io task","client":"0","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.527566099Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"io task","client":"1","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:60592)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.527607374Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"io task","client":"1","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:37525)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.52763996Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"io task","client":"2","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:58757)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.527760455Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"io task","client":"0","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.528887393Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"client":"0","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.528931347Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"client":"2","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.529529882Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.529559058Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.529775821Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"client":"1","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"[0]R flush_numbers[0..12]: [1, 22, 22, 20, 20, 19, 24, 13, 23, 20, 19, 18]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530288367Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"downstairs","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"[0]R generation[0..12]: [1, 12, 12, 11, 11, 10, 13, 6, 12, 11, 10, 10]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530320372Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"downstairs","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.53034442Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"downstairs","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"[1]R flush_numbers[0..12]: [1, 22, 22, 20, 20, 19, 24, 13, 23, 20, 19, 18]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530363607Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"downstairs","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"[1]R generation[0..12]: [1, 12, 12, 11, 11, 10, 13, 6, 12, 11, 10, 10]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530382005Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"downstairs","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530401712Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"downstairs","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"[2]R flush_numbers[0..12]: [1, 22, 22, 20, 20, 19, 24, 13, 23, 20, 19, 18]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.53042041Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"downstairs","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"[2]R generation[0..12]: [1, 12, 12, 11, 11, 10, 13, 6, 12, 11, 10, 10]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530438988Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"downstairs","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530458755Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"downstairs","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"Max found gen is 14","v":0Wait for a query_work_queue command to finish before sending IO ,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530482002Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"downstairs","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"Generation requested: 14 >= found:14","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.53049925Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"downstairs","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"Next flush: 25","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530515458Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"downstairs","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530531926Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"":"downstairs","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530548424Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530563652Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"client":"0","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.53058058Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"client":"1","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530597348Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"client":"2","":"downstairs","session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530615995Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530632193Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"fc345908-83dc-43e0-ae8c-d2c8c0b00db2 is now active with session: 62266807-fb88-4de1-b939-ad08d6ff89ec","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530650031Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530668069Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256,"session_id":"62266807-fb88-4de1-b939-ad08d6ff89ec"} {"msg":"The guest has finished waiting for activation with:14","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530684107Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256} {"msg":"Activated sub_volume fc345908-83dc-43e0-ae8c-d2c8c0b00db2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:18.530698015Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 472 len 32768 data: 3 3 3 3 3 2 3 3 03/30 Write block 585 len 24576 data: 5 6 6 8 8 7 04/30 Write block 404 len 8192 data: 2 2 05/30 Read block 2 len 28672 06/30 Write block 543 len 32768 data: 4 4 4 4 4 4 3 3 07/30 Read block 536 len 40960 08/30 Write block 235 len 8192 data: 2 2 09/30 Write block 410 len 40960 data: 4 4 3 3 3 3 3 3 3 2 10/30 Flush 11/30 Write block 349 len 12288 data: 2 2 2 12/30 Read block 131 len 32768 13/30 Flush 14/30 Write block 206 len 12288 data: 3 4 3 15/30 Read block 557 len 36864 16/30 Flush 17/30 Write block 11 len 16384 data: 2 2 2 2 18/30 Read block 241 len 20480 19/30 Write block 292 len 20480 data: 2 3 3 3 2 20/30 Read block 262 len 12288 21/30 Read block 77 len 24576 22/30 Read block 468 len 24576 23/30 Read block 84 len 4096 24/30 Write block 554 len 16384 data: 2 2 2 2 25/30 Write block 61 len 36864 data: 2 2 2 2 3 3 3 3 2 26/30 Write block 45 len 28672 data: 5 5 5 3 2 2 2 27/30 Write block 499 len 28672 data: 3 3 3 3 3 2 2 28/30 Write block 113 len 28672 data: 2 2 2 2 2 2 3 29/30 Read block 112 len 8192 30/30 Read block 427 len 36864 ---------------------------------------------------------------- Crucible gen:14 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 8 Done Done Done false 1002 Acked Write 6 Done Sent Done false 1003 Acked Write 2 Done Sent Done false 1004 Acked Read 7 Done Sent Done false 1005 Acked Write 8 Done Sent Done false 1006 Acked Read 10 Done Sent Done false 1007 Acked Write 2 Done Sent Done false 1008 Acked Write 10 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Read 8 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Read 3 Done Sent Done false 1020 Acked Read 6 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Read 1 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Write 7 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Read 9 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 0 28 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) Some(JobId(1000)) Some(JobId(1015)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:18.549266176Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1256} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 1 1 1 1 1 T F T <--- 1 020-039 12 12 12 22 22 22 F F F 2 040-059 12 12 12 22 22 22 T F T <--- 3 060-079 11 11 11 20 20 20 T F T <--- 4 080-099 11 11 11 20 20 20 F F F 5 100-119 10 10 10 19 19 19 T F T <--- 6 120-139 13 13 13 24 24 24 F F F 7 140-159 6 6 6 13 13 13 F F F 8 160-179 12 12 12 23 23 23 F F F 9 180-199 11 11 11 20 20 20 F F F 10 200-219 14 10 14 28 19 28 F F F <--- 11 220-239 14 10 14 26 18 26 F F F <--- 12 240-259 12 12 12 22 22 22 F F F 13 260-279 4 4 4 8 8 8 F F F 14 280-299 3 3 3 5 5 5 T F T <--- 15 300-319 12 12 12 23 23 23 F F F 16 320-339 12 12 12 23 23 23 F F F 17 340-359 14 12 14 27 23 27 F F F <--- 18 360-379 12 12 12 22 22 22 F F F 19 380-399 10 10 10 18 18 18 F F F 20 400-419 14 10 14 26 18 26 F F F <--- 21 420-439 13 13 13 24 24 24 F F F 22 440-459 13 13 13 24 24 24 F F F 23 460-479 14 12 14 26 22 26 F T F <--- 24 480-499 8 8 8 16 16 16 T F T <--- 25 500-519 8 8 8 16 16 16 T F T <--- 26 520-539 7 7 7 14 14 14 F F F 27 540-559 14 12 14 26 22 26 T F T <--- 28 560-579 4 4 4 8 8 8 F F F 29 580-599 14 12 14 26 22 26 F F F <--- Max gen: 14, Max flush: 28 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 7 Verifying data now No change after verify Loop: 7 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 12 12 12 22 22 22 F F F 2 040-059 15 15 15 29 29 29 F F F 3 060-079 15 15 15 29 29 29 F F F 4 080-099 11 11 11 20 20 20 F F F 5 100-119 15 15 15 29 29 29 F F F 6 120-139 13 13 13 24 24 24 F F F 7 140-159 6 6 6 13 13 13 F F F 8 160-179 12 12 12 23 23 23 F F F 9 180-199 11 11 11 20 20 20 F F F 10 200-219 14 14 14 28 28 28 F F F 11 220-239 14 14 14 26 26 26 F F F 12 240-259 12 12 12 22 22 22 F F F 13 260-279 4 4 4 8 8 8 F F F 14 280-299 15 15 15 29 29 29 F F F 15 300-319 12 12 12 23 23 23 F F F 16 320-339 12 12 12 23 23 23 F F F 17 340-359 14 14 14 27 27 27 F F F 18 360-379 12 12 12 22 22 22 F F F 19 380-399 10 10 10 18 18 18 F F F 20 400-419 14 14 14 26 26 26 F F F 21 420-439 13 13 13 24 24 24 F F F 22 440-459 13 13 13 24 24 24 F F F 23 460-479 14 14 14 26 26 26 F F F 24 480-499 15 15 15 29 29 29 F F F 25 500-519 15 15 15 29 29 29 F F F 26 520-539 7 7 7 14 14 14 F F F 27 540-559 15 15 15 29 29 29 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 14 14 14 26 26 26 F F F Max gen: 15, Max flush: 29 Begin loop 8 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:22.95670008Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.001819126Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.002529107Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.002589009Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.002609376Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.002636723Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264} {"msg":"Crucible 80c3928c-d93c-44b4-94d6-18d9973799a9 has session id: a9c54d5a-c29b-4ff1-99a2-81f634bb29fc","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.003238057Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"Upstairs opts: Upstairs UUID: 80c3928c-d93c-44b4-94d6-18d9973799a9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.003300749Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"Crucible stats registered with UUID: 80c3928c-d93c-44b4-94d6-18d9973799a9","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.003326226Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:23.003351153Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.003559937Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"io task","client":"0","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.003596592Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"io task","client":"2","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.00361489Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"io task","client":"1","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.003738664Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"task":"crutest"} {"msg":"The guest has requested activation with gen:16","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.003760751Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264} {"msg":"80c3928c-d93c-44b4-94d6-18d9973799a9 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.003832693Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.00385231Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"client":"0","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.003869438Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"client":"1","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.003896185Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"client":"2","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.003927961Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"io task","client":"2","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.003961566Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"io task","client":"0","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.003996412Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"io task","client":"1","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"ds_connection connected from Ok(127.0.0.1:44482)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.004031377Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"io task","client":"2","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"ds_connection connected from Ok(127.0.0.1:63650)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.004063833Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"io task","client":"1","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"ds_connection connected from Ok(127.0.0.1:57777)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.004095599Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"io task","client":"0","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.005231626Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"client":"1","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.005324814Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"client":"0","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.005791876Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.005887064Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.006380212Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"client":"2","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"[0]R flush_numbers[0..12]: [29, 22, 29, 29, 20, 29, 24, 13, 23, 20, 28, 26]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.006888397Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"downstairs","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"[0]R generation[0..12]: [15, 12, 15, 15, 11, 15, 13, 6, 12, 11, 14, 14]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.006921113Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"downstairs","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.006942541Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"downstairs","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"[1]R flush_numbers[0..12]: [29, 22, 29, 29, 20, 29, 24, 13, 23, 20, 28, 26]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.006961498Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"downstairs","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"[1]R generation[0..12]: [15, 12, 15, 15, 11, 15, 13, 6, 12, 11, 14, 14]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.006979856Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"downstairs","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007004863Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"downstairs","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"[2]R flush_numbers[0..12]: [29, 22, 29, 29, 20, 29, 24, 13, 23, 20, 28, 26]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007030939Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"downstairs","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"[2]R generation[0..12]: [15, 12, 15, 15, 11, 15, 13, 6, 12, 11, 14, 14]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007050397Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"downstairs","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007070505Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"downstairs","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} Wait for a query_work_queue command to finish before sending IO {"msg":"Max found gen is 16","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007097681Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"downstairs","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"Generation requested: 16 >= found:16","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007115319Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"downstairs","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"Next flush: 30","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007131217Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"downstairs","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007147595Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"":"downstairs","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007164813Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007180191Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"client":"0","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007197309Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"client":"1","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007214167Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"client":"2","":"downstairs","session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007231724Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007247872Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"80c3928c-d93c-44b4-94d6-18d9973799a9 is now active with session: a9c54d5a-c29b-4ff1-99a2-81f634bb29fc","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.00726619Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007281648Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264,"session_id":"a9c54d5a-c29b-4ff1-99a2-81f634bb29fc"} {"msg":"The guest has finished waiting for activation with:16","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007297826Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264} {"msg":"Activated sub_volume 80c3928c-d93c-44b4-94d6-18d9973799a9","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:23.007311824Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Flush 03/30 Write block 269 len 4096 data: 3 04/30 Write block 137 len 36864 data: 2 2 2 2 2 2 2 2 2 05/30 Read block 581 len 32768 06/30 Write block 43 len 24576 data: 4 5 6 6 6 4 07/30 Write block 277 len 8192 data: 2 2 08/30 Write block 133 len 40960 data: 3 3 3 3 3 3 3 3 3 3 09/30 Write block 68 len 20480 data: 4 3 2 2 2 10/30 Read block 310 len 4096 11/30 Read block 580 len 20480 12/30 Write block 401 len 8192 data: 3 2 13/30 Read block 408 len 28672 14/30 Write block 528 len 4096 data: 2 15/30 Read block 5 len 32768 16/30 Write block 525 len 36864 data: 3 2 2 3 2 2 2 2 2 17/30 Write block 287 len 4096 data: 3 18/30 Write block 228 len 24576 data: 2 2 2 2 2 2 19/30 Read block 208 len 4096 20/30 Read block 505 len 24576 21/30 Read block 28 len 8192 22/30 Write block 479 len 20480 data: 4 2 2 2 2 23/30 Read block 164 len 8192 24/30 Read block 8 len 20480 25/30 Write block 325 len 40960 data: 3 3 3 3 3 3 2 3 3 3 26/30 Write block 474 len 32768 data: 4 4 4 3 4 5 3 3 27/30 Read block 307 len 12288 28/30 Write block 338 len 24576 data: 3 4 4 3 2 2 29/30 Read block 323 len 16384 30/30 Write block 527 len 36864 data: 3 4 3 3 3 3 3 2 2 ---------------------------------------------------------------- Crucible gen:16 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 1 Done Done Done false 1003 Acked Write 9 Done Done Done false 1004 Acked Read 8 Done Done Sent false 1005 Acked Write 6 Done Done Sent false 1006 Acked Write 2 Done Done Sent false 1007 Acked Write 10 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 1 Done Done Sent false 1010 Acked Read 5 Done Done Sent false 1011 Acked Write 2 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Read 8 Done Done Sent false 1015 Acked Write 9 Done Done Sent false 1016 Acked Write 1 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Read 1 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Read 2 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Write 8 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Write 6 Done Done Sent false 1028 Acked Read 4 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 26 28 Done 27 27 2 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1001)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:23.025231877Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1264} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 12 12 12 22 22 22 F F F 2 040-059 15 15 15 29 29 29 T T F <--- 3 060-079 15 15 15 29 29 29 T T F <--- 4 080-099 11 11 11 20 20 20 F F F 5 100-119 15 15 15 29 29 29 F F F 6 120-139 13 13 13 24 24 24 T T T 7 140-159 6 6 6 13 13 13 T T T 8 160-179 12 12 12 23 23 23 F F F 9 180-199 11 11 11 20 20 20 F F F 10 200-219 14 14 14 28 28 28 F F F 11 220-239 14 14 14 26 26 26 T T F <--- 12 240-259 12 12 12 22 22 22 F F F 13 260-279 4 4 4 8 8 8 T T T 14 280-299 15 15 15 29 29 29 T T F <--- 15 300-319 12 12 12 23 23 23 F F F 16 320-339 12 12 12 23 23 23 T T F <--- 17 340-359 14 14 14 27 27 27 T T F <--- 18 360-379 12 12 12 22 22 22 F F F 19 380-399 10 10 10 18 18 18 F F F 20 400-419 14 14 14 26 26 26 T T F <--- 21 420-439 13 13 13 24 24 24 F F F 22 440-459 13 13 13 24 24 24 F F F 23 460-479 14 14 14 26 26 26 T T F <--- 24 480-499 15 15 15 29 29 29 T T F <--- 25 500-519 15 15 15 29 29 29 F F F 26 520-539 7 7 7 14 14 14 T T F <--- 27 540-559 15 15 15 29 29 29 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 14 14 14 26 26 26 F F F Max gen: 15, Max flush: 29 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 8 Verifying data now No change after verify Loop: 8 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 12 12 12 22 22 22 F F F 2 040-059 16 16 16 30 30 30 F F F 3 060-079 16 16 16 30 30 30 F F F 4 080-099 11 11 11 20 20 20 F F F 5 100-119 15 15 15 29 29 29 F F F 6 120-139 16 16 16 30 30 30 F F F 7 140-159 16 16 16 30 30 30 F F F 8 160-179 12 12 12 23 23 23 F F F 9 180-199 11 11 11 20 20 20 F F F 10 200-219 14 14 14 28 28 28 F F F 11 220-239 16 16 16 30 30 30 F F F 12 240-259 12 12 12 22 22 22 F F F 13 260-279 16 16 16 30 30 30 F F F 14 280-299 16 16 16 30 30 30 F F F 15 300-319 12 12 12 23 23 23 F F F 16 320-339 16 16 16 30 30 30 F F F 17 340-359 16 16 16 30 30 30 F F F 18 360-379 12 12 12 22 22 22 F F F 19 380-399 10 10 10 18 18 18 F F F 20 400-419 16 16 16 30 30 30 F F F 21 420-439 13 13 13 24 24 24 F F F 22 440-459 13 13 13 24 24 24 F F F 23 460-479 16 16 16 30 30 30 F F F 24 480-499 16 16 16 30 30 30 F F F 25 500-519 15 15 15 29 29 29 F F F 26 520-539 16 16 16 30 30 30 F F F 27 540-559 15 15 15 29 29 29 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 14 14 14 26 26 26 F F F Max gen: 16, Max flush: 30 Begin loop 9 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.435514374Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.480835644Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.481489002Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.481536485Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.481570271Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.481593658Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272} {"msg":"Crucible 2f8df202-5d0d-407f-ac0a-1407e72150ec has session id: c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.482152228Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"Upstairs opts: Upstairs UUID: 2f8df202-5d0d-407f-ac0a-1407e72150ec, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.482207651Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"Crucible stats registered with UUID: 2f8df202-5d0d-407f-ac0a-1407e72150ec","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.482238407Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:27.482263724Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.482500944Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"io task","client":"2","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.482538289Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"io task","client":"1","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.482559327Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"io task","client":"0","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.482718537Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"task":"crutest"} {"msg":"The guest has requested activation with gen:18","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.482740724Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272} {"msg":"2f8df202-5d0d-407f-ac0a-1407e72150ec active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.482812395Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.482832432Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"client":"0","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.48285041Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"client":"1","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.482875567Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"client":"2","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.482899864Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"io task","client":"2","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.48293092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"io task","client":"0","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.482964026Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"io task","client":"1","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"ds_connection connected from Ok(127.0.0.1:55910)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.483025458Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"io task","client":"0","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"ds_connection connected from Ok(127.0.0.1:61037)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.483063453Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"io task","client":"1","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"ds_connection connected from Ok(127.0.0.1:61465)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.483131844Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"io task","client":"2","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.484372928Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"client":"1","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.484449198Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"client":"0","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.484985461Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.485021446Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.485253897Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"client":"2","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"[0]R flush_numbers[0..12]: [29, 22, 30, 30, 20, 29, 30, 30, 23, 20, 28, 30]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.485767343Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"downstairs","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"[0]R generation[0..12]: [15, 12, 16, 16, 11, 15, 16, 16, 12, 11, 14, 16]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.485795519Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"downstairs","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.485816606Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"downstairs","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"[1]R flush_numbers[0..12]: [29, 22, 30, 30, 20, 29, 30, 30, 23, 20, 28, 30]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.485838723Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"downstairs","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"[1]R generation[0..12]: [15, 12, 16, 16, 11, 15, 16, 16, 12, 11, 14, 16]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.485857591Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"downstairs","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.485877069Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"downstairs","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"[2]R flush_numbers[0..12]: [29, 22, 30, 30, 20, 29, 30, 30, 23, 20, 28, 30]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.485895356Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"downstairs","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"[2]R generation[0..12]: [15, 12, 16, 16, 11, 15, 16, 16, 12, 11, 14, 16]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.485913404Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"downstairs","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.485938911Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"downstairs","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {Wait for a query_work_queue command to finish before sending IO "msg":"Max found gen is 17","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.485974686Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"downstairs","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"Generation requested: 18 >= found:17","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.486008062Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"downstairs","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"Next flush: 31","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.486038208Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"downstairs","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.486066915Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"":"downstairs","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.486096041Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.486123358Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"client":"0","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.486141895Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"client":"1","":"downstairs","session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.486158823Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"client":"2","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.48618292Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.486199228Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"2f8df202-5d0d-407f-ac0a-1407e72150ec is now active with session: c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.486217916Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.486233324Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272,"session_id":"c2d0bfb0-8ac0-4142-91b5-6ce6dd99cb6a"} {"msg":"The guest has finished waiting for activation with:18","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.486249112Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272} {"msg":"Activated sub_volume 2f8df202-5d0d-407f-ac0a-1407e72150ec","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:27.48626307Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 380 len 36864 data: 3 3 3 4 3 3 3 3 3 02/30 Read block 188 len 32768 03/30 Write block 176 len 32768 data: 2 2 2 3 3 4 4 4 04/30 Write block 145 len 4096 data: 3 05/30 Write block 188 len 8192 data: 3 3 06/30 Read block 17 len 20480 07/30 Read block 437 len 16384 08/30 Read block 5 len 16384 09/30 Read block 338 len 12288 10/30 Read block 548 len 32768 11/30 Read block 90 len 32768 12/30 Flush 13/30 Write block 416 len 40960 data: 4 4 4 3 2 2 2 2 3 3 14/30 Read block 196 len 4096 15/30 Write block 366 len 12288 data: 2 2 2 16/30 Flush 17/30 Write block 524 len 20480 data: 3 4 3 4 5 18/30 Write block 231 len 4096 data: 3 19/30 Flush 20/30 Write block 293 len 4096 data: 4 21/30 Write block 517 len 20480 data: 4 4 4 4 4 22/30 Read block 308 len 36864 23/30 Flush 24/30 Write block 305 len 28672 data: 2 2 4 4 4 4 4 25/30 Read block 366 len 16384 26/30 Write block 276 len 12288 data: 2 3 3 27/30 Read block 539 len 20480 28/30 Read block 103 len 40960 29/30 Read block 76 len 4096 30/30 Write block 427 len 4096 data: 4 ---------------------------------------------------------------- Crucible gen:18 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 8 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Write 2 Done Done Done false 1005 Acked Read 5 Done Done Done false 1006 Acked Read 4 Done Done Sent false 1007 Acked Read 4 Done Done Sent false 1008 Acked Read 3 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Read 1 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Write 1 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 1 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 7 Done Done Sent false 1024 Acked Read 4 Done Done Sent false 1025 Acked Write 3 Done Done Sent false 1026 Acked Read 5 Done Done Sent false 1027 Acked Read 10 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 24 26 Done 29 29 6 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:27.505295772Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1272} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 12 12 12 22 22 22 F F F 2 040-059 16 16 16 30 30 30 F F F 3 060-079 16 16 16 30 30 30 F F F 4 080-099 11 11 11 20 20 20 F F F 5 100-119 15 15 15 29 29 29 F F F 6 120-139 16 16 16 30 30 30 F F F 7 140-159 18 18 16 31 31 30 F F T <--- 8 160-179 18 18 12 31 31 23 F F T <--- 9 180-199 18 18 11 31 31 20 F F T <--- 10 200-219 14 14 14 28 28 28 F F F 11 220-239 18 18 16 33 33 30 F F F <--- 12 240-259 12 12 12 22 22 22 F F F 13 260-279 16 16 16 30 30 30 T T F <--- 14 280-299 18 18 16 34 34 30 F F F <--- 15 300-319 12 12 12 23 23 23 T T F <--- 16 320-339 16 16 16 30 30 30 F F F 17 340-359 16 16 16 30 30 30 F F F 18 360-379 18 18 12 32 32 22 F F F <--- 19 380-399 18 18 10 31 31 18 F F T <--- 20 400-419 18 18 16 32 32 30 F F F <--- 21 420-439 18 18 13 32 32 24 T T F <--- 22 440-459 13 13 13 24 24 24 F F F 23 460-479 16 16 16 30 30 30 F F F 24 480-499 16 16 16 30 30 30 F F F 25 500-519 18 18 15 34 34 29 F F F <--- 26 520-539 18 18 16 34 34 30 F F F <--- 27 540-559 15 15 15 29 29 29 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 14 14 14 26 26 26 F F F Max gen: 18, Max flush: 34 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 9 Verifying data now No change after verify Loop: 9 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 12 12 12 22 22 22 F F F 2 040-059 16 16 16 30 30 30 F F F 3 060-079 16 16 16 30 30 30 F F F 4 080-099 11 11 11 20 20 20 F F F 5 100-119 15 15 15 29 29 29 F F F 6 120-139 16 16 16 30 30 30 F F F 7 140-159 18 18 18 31 31 31 F F F 8 160-179 18 18 18 31 31 31 F F F 9 180-199 18 18 18 31 31 31 F F F 10 200-219 14 14 14 28 28 28 F F F 11 220-239 18 18 18 33 33 33 F F F 12 240-259 12 12 12 22 22 22 F F F 13 260-279 19 19 19 35 35 35 F F F 14 280-299 18 18 18 34 34 34 F F F 15 300-319 19 19 19 35 35 35 F F F 16 320-339 16 16 16 30 30 30 F F F 17 340-359 16 16 16 30 30 30 F F F 18 360-379 18 18 18 32 32 32 F F F 19 380-399 18 18 18 31 31 31 F F F 20 400-419 18 18 18 32 32 32 F F F 21 420-439 19 19 19 35 35 35 F F F 22 440-459 13 13 13 24 24 24 F F F 23 460-479 16 16 16 30 30 30 F F F 24 480-499 16 16 16 30 30 30 F F F 25 500-519 18 18 18 34 34 34 F F F 26 520-539 18 18 18 34 34 34 F F F 27 540-559 15 15 15 29 29 29 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 14 14 14 26 26 26 F F F Max gen: 19, Max flush: 35 Begin loop 10 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.905678526Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:31.949710728Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.950104849Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.981757501Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.982429256Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.98247459Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.982495888Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.982517785Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280} {"msg":"Crucible 8a8dc025-dd04-4c0f-a2f4-cb6d4db69c4f has session id: 9c016900-3827-4e42-8da6-dd1deb796e91","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.983075744Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"Upstairs opts: Upstairs UUID: 8a8dc025-dd04-4c0f-a2f4-cb6d4db69c4f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.983126388Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"Crucible stats registered with UUID: 8a8dc025-dd04-4c0f-a2f4-cb6d4db69c4f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.983153895Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:31.983174452Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.983373097Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"io task","client":"0","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.983410832Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"io task","client":"2","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.98343083Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.98366387Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"task":"crutest"} {"msg":"The guest has requested activation with gen:20","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.983699316Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280} {"msg":"8a8dc025-dd04-4c0f-a2f4-cb6d4db69c4f active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.983732572Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.98374912Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"client":"0","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.983766197Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"client":"1","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.983782975Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"client":"2","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.983799913Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"io task","client":"2","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.983816981Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"io task","client":"0","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.983834219Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"ds_connection connected from Ok(127.0.0.1:54708)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.983977931Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"io task","client":"2","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"ds_connection connected from Ok(127.0.0.1:54447)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.984024575Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"io task","client":"1","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"ds_connection connected from Ok(127.0.0.1:58196)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.98406611Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"io task","client":"0","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.98525258Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"client":"1","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.985294655Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"client":"2","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.98533351Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"client":"0","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.985818029Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.985898599Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"[0]R flush_numbers[0..12]: [29, 22, 30, 30, 20, 29, 30, 31, 31, 31, 28, 33]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986003396Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"downstairs","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"[0]R generation[0..12]: [15, 12, 16, 16, 11, 15, 16, 18, 18, 18, 14, 18]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986036922Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"downstairs","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986058959Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"downstairs","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"[1]R flush_numbers[0..12]: [29, 22, 30, 30, 20, 29, 30, 31, 31, 31, 28, 33]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986078256Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"downstairs","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"[1]R generation[0..12]: [15, 12, 16, 16, 11, 15, 16, 18, 18, 18, 14, 18]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986097674Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"downstairs","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986118351Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"downstairs","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"[2]R flush_numbers[0..12]: [29, 22, 30, 30, 20, 29, 30, 31, 31, 31, 28, 33]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986137199Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"downstairs","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"[2]R generation[0..12]: [15, 12, 16, 16, 11, 15, 16, 18, 18, 18, 14, 18]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986155407Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"downstairs","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986178264Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"downstairs","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"Max found gen is 20","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986194651Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"downstairs","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"Generation requested: 20 >= found:20","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.98621158Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"downstairs","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"Next flush: 36","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986227897Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"downstairs","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986244075Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"":"downstairs","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986261293Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"Wait for a query_work_queue command to finish before sending IO session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.98628452Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"client":"0","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986301258Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"client":"1","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986318246Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"client":"2","":"downstairs","session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986335304Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986351352Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"8a8dc025-dd04-4c0f-a2f4-cb6d4db69c4f is now active with session: 9c016900-3827-4e42-8da6-dd1deb796e91","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.98636924Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986384488Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280,"session_id":"9c016900-3827-4e42-8da6-dd1deb796e91"} {"msg":"The guest has finished waiting for activation with:20","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986400426Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280} {"msg":"Activated sub_volume 8a8dc025-dd04-4c0f-a2f4-cb6d4db69c4f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:31.986414144Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 266 len 32768 data: 2 3 3 4 3 3 3 2 03/30 Write block 418 len 20480 data: 5 4 3 3 3 04/30 Read block 383 len 16384 05/30 Read block 433 len 8192 06/30 Write block 197 len 40960 data: 2 2 2 2 2 2 2 2 2 4 07/30 Read block 221 len 28672 08/30 Read block 87 len 36864 09/30 Write block 531 len 16384 data: 4 4 4 3 10/30 Write block 63 len 12288 data: 3 3 4 11/30 Write block 420 len 36864 data: 4 4 4 3 4 4 3 5 4 12/30 Write block 317 len 40960 data: 2 2 2 2 2 2 3 3 4 4 13/30 Write block 174 len 40960 data: 2 2 3 3 3 4 4 5 5 5 14/30 Write block 508 len 16384 data: 2 2 2 2 15/30 Read block 457 len 40960 16/30 Write block 179 len 32768 data: 5 5 6 6 6 3 3 4 17/30 Read block 41 len 28672 18/30 Flush 19/30 Write block 125 len 32768 data: 3 3 4 4 4 4 3 3 20/30 Write block 429 len 24576 data: 4 4 4 3 3 5 21/30 Read block 385 len 12288 22/30 Read block 409 len 24576 23/30 Write block 596 len 12288 data: 3 2 2 24/30 Write block 587 len 4096 data: 7 25/30 Flush 26/30 Flush 27/30 Write block 64 len 36864 data: 4 5 4 4 5 4 3 3 3 28/30 Read block 263 len 4096 29/30 Read block 225 len 28672 30/30 Read block 437 len 20480 ---------------------------------------------------------------- Crucible gen:20 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 8 Done Done Done false 1002 Acked Write 5 Done Done Done false 1003 Acked Read 4 Sent Done Done false 1004 Acked Read 2 Sent Done Done false 1005 Acked Write 10 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Read 9 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Write 9 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Write 10 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Write 8 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Read 3 Sent Done Done false 1021 Acked Read 6 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 9 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Read 5 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 0 0 27 Done 2 29 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:32.006090845Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1280} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 12 12 12 22 22 22 F F F 2 040-059 16 16 16 30 30 30 F F F 3 060-079 16 20 20 30 37 37 F T T <--- 4 080-099 11 11 11 20 20 20 F F F 5 100-119 15 15 15 29 29 29 F F F 6 120-139 16 20 20 30 38 38 F F F <--- 7 140-159 18 18 18 31 31 31 F F F 8 160-179 18 20 20 31 37 37 F F F <--- 9 180-199 18 20 20 31 37 37 F F F <--- 10 200-219 14 20 20 28 37 37 F F F <--- 11 220-239 18 18 18 33 33 33 F F F 12 240-259 12 12 12 22 22 22 F F F 13 260-279 19 20 20 35 37 37 T F F <--- 14 280-299 18 18 18 34 34 34 F F F 15 300-319 19 20 20 35 37 37 F F F <--- 16 320-339 16 20 20 30 37 37 F F F <--- 17 340-359 16 16 16 30 30 30 F F F 18 360-379 18 18 18 32 32 32 F F F 19 380-399 18 18 18 31 31 31 F F F 20 400-419 18 20 20 32 37 37 T F F <--- 21 420-439 19 20 20 35 38 38 T F F <--- 22 440-459 13 13 13 24 24 24 F F F 23 460-479 16 16 16 30 30 30 F F F 24 480-499 16 16 16 30 30 30 F F F 25 500-519 18 20 20 34 37 37 F F F <--- 26 520-539 18 20 20 34 37 37 F F F <--- 27 540-559 15 15 15 29 29 29 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 14 20 20 26 38 38 F F F <--- Max gen: 20, Max flush: 38 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 10 Verifying data now No change after verify Loop: 10 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 12 12 12 22 22 22 F F F 2 040-059 16 16 16 30 30 30 F F F 3 060-079 21 21 21 39 39 39 F F F 4 080-099 11 11 11 20 20 20 F F F 5 100-119 15 15 15 29 29 29 F F F 6 120-139 20 20 20 38 38 38 F F F 7 140-159 18 18 18 31 31 31 F F F 8 160-179 20 20 20 37 37 37 F F F 9 180-199 20 20 20 37 37 37 F F F 10 200-219 20 20 20 37 37 37 F F F 11 220-239 18 18 18 33 33 33 F F F 12 240-259 12 12 12 22 22 22 F F F 13 260-279 20 20 20 37 37 37 F F F 14 280-299 18 18 18 34 34 34 F F F 15 300-319 20 20 20 37 37 37 F F F 16 320-339 20 20 20 37 37 37 F F F 17 340-359 16 16 16 30 30 30 F F F 18 360-379 18 18 18 32 32 32 F F F 19 380-399 18 18 18 31 31 31 F F F 20 400-419 20 20 20 37 37 37 F F F 21 420-439 20 20 20 38 38 38 F F F 22 440-459 13 13 13 24 24 24 F F F 23 460-479 16 16 16 30 30 30 F F F 24 480-499 16 16 16 30 30 30 F F F 25 500-519 20 20 20 37 37 37 F F F 26 520-539 20 20 20 37 37 37 F F F 27 540-559 15 15 15 29 29 29 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 20 20 20 38 38 38 F F F Max gen: 21, Max flush: 39 Begin loop 11 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.43575632Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:36.480579243Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.480951326Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.512971692Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.51354291Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.513590824Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.513606922Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.513628969Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288} {"msg":"Crucible ebd206f3-fc48-4ca5-8cd6-ee373b7f076d has session id: 874c58a1-30f0-47de-a389-d07467b57343","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.514248311Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"Upstairs opts: Upstairs UUID: ebd206f3-fc48-4ca5-8cd6-ee373b7f076d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.514294396Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"Crucible stats registered with UUID: ebd206f3-fc48-4ca5-8cd6-ee373b7f076d","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.514314643Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:36.514330641Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.514525806Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"io task","client":"0","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.514562202Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.514580669Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"io task","client":"2","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.514757907Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"task":"crutest"} {"msg":"The guest has requested activation with gen:22","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.514781254Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288} {"msg":"ebd206f3-fc48-4ca5-8cd6-ee373b7f076d active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.514856385Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.514877212Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"client":"0","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.51489457Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"client":"1","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.514914597Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"client":"2","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.514943264Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"io task","client":"2","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.51497528Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"io task","client":"0","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.515007046Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"ds_connection connected from Ok(127.0.0.1:35961)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.515142129Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"io task","client":"1","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"ds_connection connected from Ok(127.0.0.1:57513)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.515256024Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"io task","client":"2","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"ds_connection connected from Ok(127.0.0.1:35386)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.515282471Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"io task","client":"0","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.516472441Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"client":"1","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.516507267Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"client":"2","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.516528934Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"client":"0","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517073805Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517103252Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"[0]R flush_numbers[0..12]: [29, 22, 30, 39, 20, 29, 38, 31, 37, 37, 37, 33]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517126119Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"downstairs","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"[0]R generation[0..12]: [15, 12, 16, 21, 11, 15, 20, 18, 20, 20, 20, 18]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517146026Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"downstairs","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517166024Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"downstairs","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"[1]R flush_numbers[0..12]: [29, 22, 30, 39, 20, 29, 38, 31, 37, 37, 37, 33]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517184711Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"downstairs","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"[1]R generation[0..12]: [15, 12, 16, 21, 11, 15, 20, 18, 20, 20, 20, 18]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517202779Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"downstairs","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517222317Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"downstairs","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"[2]R flush_numbers[0..12]: [29, 22, 30, 39, 20, 29, 38, 31, 37, 37, 37, 33]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517240884Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"downstairs","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"[2]R generation[0..12]: [15, 12, 16, 21, 11, 15, 20, 18, 20, 20, 20, 18]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517259292Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"downstairs","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517279Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"downstairs","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"Max found gen is 22","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517295627Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"downstairs","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"Wait for a query_work_queue command to finish before sending IO Generation requested: 22 >= found:22","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517318475Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"downstairs","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"Next flush: 40","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517334593Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"downstairs","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.51735078Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"":"downstairs","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517367698Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517384596Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"client":"0","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517402584Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"client":"1","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517419232Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"client":"2","":"downstairs","session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.51743602Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517452068Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"ebd206f3-fc48-4ca5-8cd6-ee373b7f076d is now active with session: 874c58a1-30f0-47de-a389-d07467b57343","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517469885Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517484983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288,"session_id":"874c58a1-30f0-47de-a389-d07467b57343"} {"msg":"The guest has finished waiting for activation with:22","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.517500892Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288} {"msg":"Activated sub_volume ebd206f3-fc48-4ca5-8cd6-ee373b7f076d","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:36.51751496Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 243 len 16384 data: 3 3 3 3 02/30 Write block 362 len 32768 data: 2 2 2 2 3 3 3 2 03/30 Write block 157 len 12288 data: 2 2 3 04/30 Read block 566 len 36864 05/30 Read block 260 len 32768 06/30 Write block 392 len 4096 data: 2 07/30 Write block 596 len 8192 data: 4 3 08/30 Read block 567 len 20480 09/30 Read block 221 len 32768 10/30 Write block 239 len 4096 data: 2 11/30 Read block 56 len 36864 12/30 Read block 99 len 36864 13/30 Read block 11 len 40960 14/30 Write block 286 len 40960 data: 3 4 2 2 2 2 3 5 4 4 15/30 Read block 255 len 32768 16/30 Read block 109 len 24576 17/30 Write block 144 len 8192 data: 3 4 18/30 Read block 99 len 36864 19/30 Read block 344 len 8192 20/30 Read block 166 len 4096 21/30 Read block 356 len 24576 22/30 Flush 23/30 Write block 408 len 24576 data: 3 4 5 5 4 4 24/30 Read block 334 len 20480 25/30 Read block 286 len 24576 26/30 Read block 300 len 40960 27/30 Read block 274 len 36864 28/30 Read block 425 len 24576 29/30 Read block 366 len 40960 30/30 Read block 27 len 4096 ---------------------------------------------------------------- Crucible gen:22 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Read 9 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Write 1 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Read 5 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Write 1 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Read 10 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Write 2 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 6 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Read 1 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 0 28 Done 2 30 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:36.536949711Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1288} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 12 12 12 22 22 22 F F F 2 040-059 16 16 16 30 30 30 F F F 3 060-079 21 21 21 39 39 39 F F F 4 080-099 11 11 11 20 20 20 F F F 5 100-119 15 15 15 29 29 29 F F F 6 120-139 20 20 20 38 38 38 F F F 7 140-159 18 22 22 31 40 40 F F F <--- 8 160-179 20 20 20 37 37 37 F F F 9 180-199 20 20 20 37 37 37 F F F 10 200-219 20 20 20 37 37 37 F F F 11 220-239 18 22 22 33 40 40 F F F <--- 12 240-259 12 22 22 22 40 40 T F F <--- 13 260-279 20 20 20 37 37 37 F F F 14 280-299 18 22 22 34 40 40 F F F <--- 15 300-319 20 20 20 37 37 37 F F F 16 320-339 20 20 20 37 37 37 F F F 17 340-359 16 16 16 30 30 30 F F F 18 360-379 18 22 22 32 40 40 T F F <--- 19 380-399 18 22 22 31 40 40 F F F <--- 20 400-419 20 20 20 37 37 37 F T T <--- 21 420-439 20 20 20 38 38 38 F F F 22 440-459 13 13 13 24 24 24 F F F 23 460-479 16 16 16 30 30 30 F F F 24 480-499 16 16 16 30 30 30 F F F 25 500-519 20 20 20 37 37 37 F F F 26 520-539 20 20 20 37 37 37 F F F 27 540-559 15 15 15 29 29 29 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 20 22 22 38 40 40 F F F <--- Max gen: 22, Max flush: 40 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 11 Verifying data now No change after verify Loop: 11 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 12 12 12 22 22 22 F F F 2 040-059 16 16 16 30 30 30 F F F 3 060-079 21 21 21 39 39 39 F F F 4 080-099 11 11 11 20 20 20 F F F 5 100-119 15 15 15 29 29 29 F F F 6 120-139 20 20 20 38 38 38 F F F 7 140-159 22 22 22 40 40 40 F F F 8 160-179 20 20 20 37 37 37 F F F 9 180-199 20 20 20 37 37 37 F F F 10 200-219 20 20 20 37 37 37 F F F 11 220-239 22 22 22 40 40 40 F F F 12 240-259 22 22 22 40 40 40 F F F 13 260-279 20 20 20 37 37 37 F F F 14 280-299 22 22 22 40 40 40 F F F 15 300-319 20 20 20 37 37 37 F F F 16 320-339 20 20 20 37 37 37 F F F 17 340-359 16 16 16 30 30 30 F F F 18 360-379 22 22 22 40 40 40 F F F 19 380-399 22 22 22 40 40 40 F F F 20 400-419 23 23 23 41 41 41 F F F 21 420-439 20 20 20 38 38 38 F F F 22 440-459 13 13 13 24 24 24 F F F 23 460-479 16 16 16 30 30 30 F F F 24 480-499 16 16 16 30 30 30 F F F 25 500-519 20 20 20 37 37 37 F F F 26 520-539 20 20 20 37 37 37 F F F 27 540-559 15 15 15 29 29 29 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 22 22 22 40 40 40 F F F Max gen: 23, Max flush: 41 Begin loop 12 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.950225281Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.996490602Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.997092326Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.99713873Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.997161417Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.997184085Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296} {"msg":"Crucible 5ca81f68-3df1-4f1f-a3a5-16fbd815d5e1 has session id: 632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.997728846Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"Upstairs opts: Upstairs UUID: 5ca81f68-3df1-4f1f-a3a5-16fbd815d5e1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.997876907Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"Crucible stats registered with UUID: 5ca81f68-3df1-4f1f-a3a5-16fbd815d5e1","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.997915033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:40.99793296Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.998106928Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"io task","client":"0","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.998145424Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"io task","client":"2","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.99816839Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"io task","client":"1","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.998348998Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"task":"crutest"} {"msg":"The guest has requested activation with gen:24","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.998382663Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296} {"msg":"5ca81f68-3df1-4f1f-a3a5-16fbd815d5e1 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.998439866Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.998470772Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"client":"0","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.99849Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"client":"1","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.998507418Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"client":"2","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.998524356Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"io task","client":"2","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.998541563Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"io task","client":"0","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.99856911Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"io task","client":"1","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"ds_connection connected from Ok(127.0.0.1:63003)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.998676557Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"io task","client":"2","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"ds_connection connected from Ok(127.0.0.1:39248)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.99872774Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"io task","client":"0","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"ds_connection connected from Ok(127.0.0.1:41632)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:40.998843125Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"io task","client":"1","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.000002339Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"client":"1","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.000044614Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"client":"2","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.00007478Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"client":"0","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.000604024Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.000708741Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"[0]R flush_numbers[0..12]: [29, 22, 30, 39, 20, 29, 38, 40, 37, 37, 37, 40]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.000816567Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"downstairs","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"[0]R generation[0..12]: [15, 12, 16, 21, 11, 15, 20, 22, 20, 20, 20, 22]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.000842884Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"downstairs","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.000864141Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"downstairs","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"[1]R flush_numbers[0..12]: [29, 22, 30, 39, 20, 29, 38, 40, 37, 37, 37, 40]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.000883958Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"downstairs","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"[1]R generation[0..12]: [15, 12, 16, 21, 11, 15, 20, 22, 20, 20, 20, 22]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.000902666Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"downstairs","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.000922294Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"downstairs","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"[2]R flush_numbers[0..12]: [29, 22, 30, 39, 20, 29, 38, 40, 37, 37, 37, 40]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.000941051Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"downstairs","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"[2]R generation[0..12]: [15, 12, 16, 21, 11, 15, 20, 22, 20, 20, 20, 22]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.000963768Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"downstairs","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.000983866Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"downstairs","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"Max found gen is 24","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001000554Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"downstairs","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"Generation requested: 24 >= found:24","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001017762Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"downstairs","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"Next flush: 42","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001042028Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"downstairs","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001059856Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"":"downstairs","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001076854Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758"Wait for a query_work_queue command to finish before sending IO ,"pid":1296,"session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001099901Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"client":"0","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001116849Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"client":"1","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001133447Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"client":"2","":"downstairs","session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001150545Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001166733Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"5ca81f68-3df1-4f1f-a3a5-16fbd815d5e1 is now active with session: 632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001184581Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001199729Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296,"session_id":"632f3ddc-f0a0-4a2b-b5c1-accd5cd33bcf"} {"msg":"The guest has finished waiting for activation with:24","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001215757Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296} {"msg":"Activated sub_volume 5ca81f68-3df1-4f1f-a3a5-16fbd815d5e1","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:41.001229565Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 250 len 20480 data: 4 3 3 3 3 02/30 Read block 148 len 36864 03/30 Write block 355 len 8192 data: 2 2 04/30 Read block 375 len 8192 05/30 Flush 06/30 Flush 07/30 Write block 204 len 8192 data: 3 3 08/30 Read block 99 len 40960 09/30 Write block 249 len 16384 data: 3 5 4 4 10/30 Read block 87 len 24576 11/30 Read block 259 len 8192 12/30 Write block 82 len 36864 data: 3 3 3 3 3 3 3 3 2 13/30 Flush 14/30 Write block 196 len 40960 data: 2 3 3 3 3 3 3 3 4 4 15/30 Read block 476 len 36864 16/30 Read block 114 len 28672 17/30 Write block 542 len 32768 data: 5 5 5 5 5 5 5 4 18/30 Flush 19/30 Write block 423 len 28672 data: 4 5 5 4 6 5 5 20/30 Flush 21/30 Write block 276 len 40960 data: 3 4 4 3 2 2 2 3 3 3 22/30 Write block 179 len 8192 data: 6 6 23/30 Write block 203 len 20480 data: 4 5 5 5 5 24/30 Flush 25/30 Flush 26/30 Write block 255 len 16384 data: 3 3 3 4 27/30 Read block 400 len 24576 28/30 Read block 546 len 40960 29/30 Write block 151 len 32768 data: 2 2 2 2 2 2 3 3 30/30 Write block 89 len 36864 data: 4 3 2 2 2 2 3 3 4 ---------------------------------------------------------------- Crucible gen:24 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Write 2 Done Done Done false 1003 Acked Read 2 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 2 Done Done Sent false 1007 Acked Read 10 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 10 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Write 8 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 7 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Write 5 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Read 6 Done Done Sent false 1027 Acked Read 10 Done Done Sent false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 27 31 Done 28 28 3 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:41.020805209Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1296} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 12 12 12 22 22 22 F F F 2 040-059 16 16 16 30 30 30 F F F 3 060-079 21 21 21 39 39 39 F F F 4 080-099 24 24 11 44 44 20 T T F <--- 5 100-119 15 15 15 29 29 29 F F F 6 120-139 20 20 20 38 38 38 F F F 7 140-159 22 22 22 40 40 40 T T F <--- 8 160-179 24 24 20 47 47 37 F F F <--- 9 180-199 24 24 20 47 47 37 F F F <--- 10 200-219 24 24 20 47 47 37 F F F <--- 11 220-239 22 22 22 40 40 40 F F F 12 240-259 24 24 22 44 44 40 T T T <--- 13 260-279 24 24 20 47 47 37 F F F <--- 14 280-299 24 24 22 47 47 40 F F F <--- 15 300-319 20 20 20 37 37 37 F F F 16 320-339 20 20 20 37 37 37 F F F 17 340-359 24 24 16 42 42 30 F F T <--- 18 360-379 22 22 22 40 40 40 F F F 19 380-399 22 22 22 40 40 40 F F F 20 400-419 23 23 23 41 41 41 F F F 21 420-439 24 24 20 46 46 38 F F F <--- 22 440-459 13 13 13 24 24 24 F F F 23 460-479 16 16 16 30 30 30 F F F 24 480-499 16 16 16 30 30 30 F F F 25 500-519 20 20 20 37 37 37 F F F 26 520-539 20 20 20 37 37 37 F F F 27 540-559 24 24 15 45 45 29 F F F <--- 28 560-579 4 4 4 8 8 8 F F F 29 580-599 22 22 22 40 40 40 F F F Max gen: 24, Max flush: 47 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 12 Verifying data now No change after verify Loop: 12 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 12 12 12 22 22 22 F F F 2 040-059 16 16 16 30 30 30 F F F 3 060-079 21 21 21 39 39 39 F F F 4 080-099 25 25 25 48 48 48 F F F 5 100-119 15 15 15 29 29 29 F F F 6 120-139 20 20 20 38 38 38 F F F 7 140-159 25 25 25 48 48 48 F F F 8 160-179 24 24 24 47 47 47 F F F 9 180-199 24 24 24 47 47 47 F F F 10 200-219 24 24 24 47 47 47 F F F 11 220-239 22 22 22 40 40 40 F F F 12 240-259 25 25 25 48 48 48 F F F 13 260-279 24 24 24 47 47 47 F F F 14 280-299 24 24 24 47 47 47 F F F 15 300-319 20 20 20 37 37 37 F F F 16 320-339 20 20 20 37 37 37 F F F 17 340-359 24 24 24 42 42 42 F F F 18 360-379 22 22 22 40 40 40 F F F 19 380-399 22 22 22 40 40 40 F F F 20 400-419 23 23 23 41 41 41 F F F 21 420-439 24 24 24 46 46 46 F F F 22 440-459 13 13 13 24 24 24 F F F 23 460-479 16 16 16 30 30 30 F F F 24 480-499 16 16 16 30 30 30 F F F 25 500-519 20 20 20 37 37 37 F F F 26 520-539 20 20 20 37 37 37 F F F 27 540-559 24 24 24 45 45 45 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 22 22 22 40 40 40 F F F Max gen: 25, Max flush: 48 Begin loop 13 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.414024476Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.45939088Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.460044978Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.460093921Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.460110109Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.460132476Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304} {"msg":"Crucible 3cb111fd-5b5a-4f4a-9c77-bc5162481831 has session id: c4a2c65d-eb31-4e14-bafb-7b9221376c81","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.46065919Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"Upstairs opts: Upstairs UUID: 3cb111fd-5b5a-4f4a-9c77-bc5162481831, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.460717053Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"Crucible stats registered with UUID: 3cb111fd-5b5a-4f4a-9c77-bc5162481831","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.460749449Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:45.460777375Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.461004686Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"io task","client":"0","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.46105254Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"io task","client":"1","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.461082627Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"io task","client":"2","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.461236267Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"task":"crutest"} {"msg":"The guest has requested activation with gen:26","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.461266593Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304} {"msg":"3cb111fd-5b5a-4f4a-9c77-bc5162481831 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.461331225Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.461355182Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"client":"0","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.46137348Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"client":"1","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.461390768Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"client":"2","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.461407466Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"io task","client":"2","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.461424434Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"io task","client":"0","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.461483526Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"io task","client":"1","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"ds_connection connected from Ok(127.0.0.1:49832)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.461513173Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"io task","client":"2","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"ds_connection connected from Ok(127.0.0.1:60641)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.461726136Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"io task","client":"0","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"ds_connection connected from Ok(127.0.0.1:62926)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.461859759Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"io task","client":"1","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.462766724Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"client":"1","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.462814358Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"client":"0","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.463410683Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.463446669Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.463706926Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"client":"2","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"[0]R flush_numbers[0..12]: [29, 22, 30, 39, 48, 29, 38, 48, 47, 47, 47, 40]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464217042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"downstairs","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"[0]R generation[0..12]: [15, 12, 16, 21, 25, 15, 20, 25, 24, 24, 24, 22]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464245658Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"downstairs","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464267346Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"downstairs","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"[1]R flush_numbers[0..12]: [29, 22, 30, 39, 48, 29, 38, 48, 47, 47, 47, 40]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464286233Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"downstairs","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"[1]R generation[0..12]: [15, 12, 16, 21, 25, 15, 20, 25, 24, 24, 24, 22]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464304611Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"downstairs","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464324208Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"downstairs","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"[2]R flush_numbers[0..12]: [29, 22, 30, 39, 48, 29, 38, 48, 47, 47, 47, 40]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464342536Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"downstairs","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"[2]R generation[0..12]: [15, 12, 16, 21, 25, 15, 20, 25, 24, 24, 24, 22]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464360984Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"downstairs","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464381361Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"downstairs","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"Max found gen is 26","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464397389Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"downstairs","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"Generation requested: 26 >= found:26","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-11-06T20:11:45.464414277Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"downstairs","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"Next flush: 49","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464434844Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"downstairs","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464451802Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"":"downstairs","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.46446903Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464484398Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"client":"0","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464503236Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"client":"1","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464521063Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"client":"2","":"downstairs","session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464537891Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464553759Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"3cb111fd-5b5a-4f4a-9c77-bc5162481831 is now active with session: c4a2c65d-eb31-4e14-bafb-7b9221376c81","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464571597Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"Set Active after no reconciliation","v":0,"name":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 crucible","level":30,"time":"2025-11-06T20:11:45.464590885Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304,"session_id":"c4a2c65d-eb31-4e14-bafb-7b9221376c81"} {"msg":"The guest has finished waiting for activation with:26","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464607063Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304} {"msg":"Activated sub_volume 3cb111fd-5b5a-4f4a-9c77-bc5162481831","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:45.464621041Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 350 len 32768 data: 3 3 2 2 2 3 3 2 02/30 Flush 03/30 Write block 448 len 36864 data: 2 2 2 3 3 4 4 4 4 04/30 Flush 05/30 Write block 265 len 12288 data: 2 3 4 06/30 Write block 462 len 16384 data: 2 2 2 2 07/30 Read block 63 len 36864 08/30 Read block 491 len 12288 09/30 Read block 163 len 8192 10/30 Write block 361 len 4096 data: 2 11/30 Read block 61 len 16384 12/30 Write block 554 len 24576 data: 3 3 3 3 2 2 13/30 Read block 476 len 40960 14/30 Read block 428 len 16384 15/30 Write block 136 len 12288 data: 4 4 4 16/30 Write block 107 len 40960 data: 2 2 2 2 2 2 3 3 3 3 17/30 Write block 320 len 32768 data: 3 3 3 4 4 5 5 4 18/30 Read block 583 len 40960 19/30 Read block 189 len 20480 20/30 Read block 244 len 28672 21/30 Write block 128 len 36864 data: 5 5 5 4 4 4 4 4 5 22/30 Write block 551 len 24576 data: 2 2 2 4 4 4 23/30 Read block 262 len 36864 24/30 Write block 344 len 4096 data: 2 25/30 Write block 22 len 40960 data: 3 3 3 4 3 2 2 2 2 2 26/30 Read block 4 len 40960 27/30 Write block 142 len 12288 data: 4 3 4 28/30 Write block 116 len 24576 data: 4 3 3 4 4 4 29/30 Write block 465 len 16384 data: 3 2 3 3 30/30 Write block 450 len 32768 data: 3 4 4 5 5 5 5 5 ---------------------------------------------------------------- Crucible gen:26 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 9 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 3 Done Done Sent false 1005 Acked Write 4 Done Done Sent false 1006 Acked Read 9 Done Done Sent false 1007 Acked Read 3 Done Done Sent false 1008 Acked Read 2 Done Done Sent false 1009 Acked Write 1 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Read 4 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Write 8 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Write 9 Done Done Sent false 1021 Acked Write 6 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Write 1 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Read 10 Done Done Sent false 1026 Acked Write 3 Done Done Sent false 1027 Acked Write 6 Done Done Sent false 1028 Acked Write 4 Done Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 2 28 31 Done 27 26 0 53 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:45.483325734Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1304} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 12 12 12 22 22 22 T T F <--- 2 040-059 16 16 16 30 30 30 F F F 3 060-079 21 21 21 39 39 39 F F F 4 080-099 25 25 25 48 48 48 F F F 5 100-119 15 15 15 29 29 29 T T F <--- 6 120-139 20 20 20 38 38 38 T T F <--- 7 140-159 25 25 25 48 48 48 T T F <--- 8 160-179 24 24 24 47 47 47 F F F 9 180-199 24 24 24 47 47 47 F F F 10 200-219 24 24 24 47 47 47 F F F 11 220-239 22 22 22 40 40 40 F F F 12 240-259 25 25 25 48 48 48 F F F 13 260-279 24 24 24 47 47 47 T T F <--- 14 280-299 24 24 24 47 47 47 F F F 15 300-319 20 20 20 37 37 37 F F F 16 320-339 20 20 20 37 37 37 T T F <--- 17 340-359 26 26 26 49 49 49 T T F <--- 18 360-379 22 22 22 40 40 40 T T F <--- 19 380-399 22 22 22 40 40 40 F F F 20 400-419 23 23 23 41 41 41 F F F 21 420-439 24 24 24 46 46 46 F F F 22 440-459 26 26 13 50 50 24 T T F <--- 23 460-479 16 16 16 30 30 30 T T F <--- 24 480-499 16 16 16 30 30 30 F F F 25 500-519 20 20 20 37 37 37 F F F 26 520-539 20 20 20 37 37 37 F F F 27 540-559 24 24 24 45 45 45 T T F <--- 28 560-579 4 4 4 8 8 8 F F F 29 580-599 22 22 22 40 40 40 F F F Max gen: 26, Max flush: 50 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 13 Verifying data now No change after verify Loop: 13 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 27 27 27 51 51 51 F F F 2 040-059 16 16 16 30 30 30 F F F 3 060-079 21 21 21 39 39 39 F F F 4 080-099 25 25 25 48 48 48 F F F 5 100-119 27 27 27 51 51 51 F F F 6 120-139 27 27 27 51 51 51 F F F 7 140-159 27 27 27 51 51 51 F F F 8 160-179 24 24 24 47 47 47 F F F 9 180-199 24 24 24 47 47 47 F F F 10 200-219 24 24 24 47 47 47 F F F 11 220-239 22 22 22 40 40 40 F F F 12 240-259 25 25 25 48 48 48 F F F 13 260-279 27 27 27 51 51 51 F F F 14 280-299 24 24 24 47 47 47 F F F 15 300-319 20 20 20 37 37 37 F F F 16 320-339 27 27 27 51 51 51 F F F 17 340-359 27 27 27 51 51 51 F F F 18 360-379 27 27 27 51 51 51 F F F 19 380-399 22 22 22 40 40 40 F F F 20 400-419 23 23 23 41 41 41 F F F 21 420-439 24 24 24 46 46 46 F F F 22 440-459 27 27 27 51 51 51 F F F 23 460-479 27 27 27 51 51 51 F F F 24 480-499 16 16 16 30 30 30 F F F 25 500-519 20 20 20 37 37 37 F F F 26 520-539 20 20 20 37 37 37 F F F 27 540-559 27 27 27 51 51 51 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 22 22 22 40 40 40 F F F Max gen: 27, Max flush: 51 Begin loop 14 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.883893965Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:49.928105385Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.928508914Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.959880922Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.960519671Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.960568285Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.960584293Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.96060652Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312} {"msg":"Crucible 2b8a9cfd-f61d-458c-b46d-57077b1c1308 has session id: b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961159561Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"Upstairs opts: Upstairs UUID: 2b8a9cfd-f61d-458c-b46d-57077b1c1308, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961203255Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"Crucible stats registered with UUID: 2b8a9cfd-f61d-458c-b46d-57077b1c1308","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961222483Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:49.961238491Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961458023Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"io task","client":"0","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961506407Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"io task","client":"2","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961538013Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"io task","client":"1","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961705642Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"task":"crutest"} {"msg":"The guest has requested activation with gen:28","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961734728Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312} {"msg":"2b8a9cfd-f61d-458c-b46d-57077b1c1308 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.96179805Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961819797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"client":"0","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961837675Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"client":"1","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961855723Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"client":"2","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961872761Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"io task","client":"2","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961889829Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"io task","client":"0","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961906886Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"io task","client":"1","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"ds_connection connected from Ok(127.0.0.1:39602)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.961965949Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"io task","client":"2","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"ds_connection connected from Ok(127.0.0.1:48726)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.962018303Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"io task","client":"0","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"ds_connection connected from Ok(127.0.0.1:53309)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.962054588Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"io task","client":"1","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.963363943Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"client":"1","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.963406737Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"client":"2","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.963429105Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"client":"0","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.963933231Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964029459Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"[0]R flush_numbers[0..12]: [29, 51, 30, 39, 48, 51, 51, 51, 47, 47, 47, 40]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964108779Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"downstairs","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"[0]R generation[0..12]: [15, 27, 16, 21, 25, 27, 27, 27, 24, 24, 24, 22]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964132616Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"downstairs","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964152923Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"downstairs","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"[1]R flush_numbers[0..12]: [29, 51, 30, 39, 48, 51, 51, 51, 47, 47, 47, 40]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964171431Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"downstairs","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"[1]R generation[0..12]: [15, 27, 16, 21, 25, 27, 27, 27, 24, 24, 24, 22]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964189979Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"downstairs","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964209646Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"downstairs","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"[2]R flush_numbers[0..12]: [29, 51, 30, 39, 48, 51, 51, 51, 47, 47, 47, 40]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964230994Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"downstairs","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"[2]R generation[0..12]: [15, 27, 16, 21, 25, 27, 27, 27, 24, 24, 24, 22]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964249451Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"downstairs","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964275068Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"downstairs","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"Max found gen is 28","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964304994Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"downstairs","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"Generation requested: 28 >= found:28","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964334331Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"downstairs",""Wait for a query_work_queue command to finish before sending IO :"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"Next flush: 52","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964371326Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"downstairs","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964401302Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"":"downstairs","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.96442077Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964443487Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"client":"0","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964473353Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"client":"1","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964504339Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"client":"2","":"downstairs","session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964523617Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964548914Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"2b8a9cfd-f61d-458c-b46d-57077b1c1308 is now active with session: b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.96458124Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964603927Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312,"session_id":"b7eb8fd8-db6b-4de6-8b13-b82cd5e1e66b"} {"msg":"The guest has finished waiting for activation with:28","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:49.964623604Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312} {"msg":"Activated sub_volume 2b8a9cfd-f61d-458c-b46d-57077b1c1308","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:11:49.964650801Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 279 len 12288 data: 4 3 3 03/30 Flush 04/30 Write block 222 len 40960 data: 2 3 2 2 2 2 3 3 3 4 05/30 Read block 453 len 32768 06/30 Flush 07/30 Write block 553 len 24576 data: 3 5 5 5 4 3 08/30 Read block 546 len 32768 09/30 Flush 10/30 Flush 11/30 Write block 361 len 24576 data: 3 3 3 3 3 4 12/30 Write block 53 len 8192 data: 2 2 13/30 Read block 113 len 16384 14/30 Read block 259 len 12288 15/30 Read block 438 len 40960 16/30 Flush 17/30 Write block 548 len 32768 data: 6 5 4 3 3 4 6 6 18/30 Flush 19/30 Write block 384 len 8192 data: 4 4 20/30 Read block 10 len 32768 21/30 Read block 445 len 40960 22/30 Write block 358 len 8192 data: 2 2 23/30 Read block 458 len 16384 24/30 Flush 25/30 Write block 309 len 32768 data: 5 5 5 3 4 4 2 2 26/30 Flush 27/30 Write block 464 len 32768 data: 3 4 3 4 4 3 3 3 28/30 Read block 107 len 12288 29/30 Write block 424 len 28672 data: 6 6 5 7 6 6 5 30/30 Write block 25 len 12288 data: 5 4 3 ---------------------------------------------------------------- Crucible gen:28 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Sent Done Done false 1001 Acked Write 3 Sent Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 10 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 6 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 6 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Read 4 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 8 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Read 8 Sent Done Done false 1020 Acked Read 10 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 8 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Read 3 Sent Done Done false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:49.984534446Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1312} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 27 27 27 51 51 51 F T T <--- 2 040-059 16 28 28 30 57 57 F F F <--- 3 060-079 21 21 21 39 39 39 F F F 4 080-099 25 25 25 48 48 48 F F F 5 100-119 27 27 27 51 51 51 F F F 6 120-139 27 27 27 51 51 51 F F F 7 140-159 27 27 27 51 51 51 F F F 8 160-179 24 24 24 47 47 47 F F F 9 180-199 24 24 24 47 47 47 F F F 10 200-219 24 24 24 47 47 47 F F F 11 220-239 22 28 28 40 54 54 F F F <--- 12 240-259 25 25 25 48 48 48 F F F 13 260-279 27 28 28 51 53 53 F F F <--- 14 280-299 24 28 28 47 53 53 F F F <--- 15 300-319 20 28 28 37 60 60 F F F <--- 16 320-339 27 27 27 51 51 51 F F F 17 340-359 27 28 28 51 59 59 F F F <--- 18 360-379 27 28 28 51 57 57 F F F <--- 19 380-399 22 28 28 40 59 59 F F F <--- 20 400-419 23 23 23 41 41 41 F F F 21 420-439 24 24 24 46 46 46 F T T <--- 22 440-459 27 27 27 51 51 51 F F F 23 460-479 27 27 27 51 51 51 F T T <--- 24 480-499 16 16 16 30 30 30 F F F 25 500-519 20 20 20 37 37 37 F F F 26 520-539 20 20 20 37 37 37 F F F 27 540-559 27 28 28 51 58 58 F F F <--- 28 560-579 4 4 4 8 8 8 F F F 29 580-599 22 22 22 40 40 40 F F F Max gen: 28, Max flush: 60 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 14 Verifying data now No change after verify Loop: 14 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 29 29 29 61 61 61 F F F 2 040-059 28 28 28 57 57 57 F F F 3 060-079 21 21 21 39 39 39 F F F 4 080-099 25 25 25 48 48 48 F F F 5 100-119 27 27 27 51 51 51 F F F 6 120-139 27 27 27 51 51 51 F F F 7 140-159 27 27 27 51 51 51 F F F 8 160-179 24 24 24 47 47 47 F F F 9 180-199 24 24 24 47 47 47 F F F 10 200-219 24 24 24 47 47 47 F F F 11 220-239 28 28 28 54 54 54 F F F 12 240-259 25 25 25 48 48 48 F F F 13 260-279 28 28 28 53 53 53 F F F 14 280-299 28 28 28 53 53 53 F F F 15 300-319 28 28 28 60 60 60 F F F 16 320-339 27 27 27 51 51 51 F F F 17 340-359 28 28 28 59 59 59 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 28 28 28 59 59 59 F F F 20 400-419 23 23 23 41 41 41 F F F 21 420-439 29 29 29 61 61 61 F F F 22 440-459 27 27 27 51 51 51 F F F 23 460-479 29 29 29 61 61 61 F F F 24 480-499 16 16 16 30 30 30 F F F 25 500-519 20 20 20 37 37 37 F F F 26 520-539 20 20 20 37 37 37 F F F 27 540-559 28 28 28 58 58 58 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 22 22 22 40 40 40 F F F Max gen: 29, Max flush: 61 Begin loop 15 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.413939384Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.460475381Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.461113651Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.461172943Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.461193241Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.461220887Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320} {"msg":"Crucible 4883eb6e-0c14-40d1-b8c9-d7685b1d81f5 has session id: 0915cbf0-3c32-4597-a215-2db7a1be014f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.461767618Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"Upstairs opts: Upstairs UUID: 4883eb6e-0c14-40d1-b8c9-d7685b1d81f5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.461824181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"Crucible stats registered with UUID: 4883eb6e-0c14-40d1-b8c9-d7685b1d81f5","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.461847908Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:54.461864956Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462127653Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462177237Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462210033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462365833Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"task":"crutest"} {"msg":"The guest has requested activation with gen:30","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462393289Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320} {"msg":"4883eb6e-0c14-40d1-b8c9-d7685b1d81f5 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462452432Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462490217Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"client":"0","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462524053Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"client":"1","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462557509Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"client":"2","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462588365Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462619221Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462842483Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"ds_connection connected from Ok(127.0.0.1:35613)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462886367Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"io task","client":"2","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"ds_connection connected from Ok(127.0.0.1:60618)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462921923Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"io task","client":"0","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"ds_connection connected from Ok(127.0.0.1:61764)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.462952809Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"io task","client":"1","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.463899Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"client":"0","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.463941984Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"client":"2","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.464266923Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"client":"1","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.464492855Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.46452945Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"[0]R flush_numbers[0..12]: [29, 61, 57, 39, 48, 51, 51, 51, 47, 47, 47, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.464827903Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"downstairs","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"[0]R generation[0..12]: [15, 29, 28, 21, 25, 27, 27, 27, 24, 24, 24, 28]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.464854659Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"downstairs","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.464875377Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"downstairs","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"[1]R flush_numbers[0..12]: [29, 61, 57, 39, 48, 51, 51, 51, 47, 47, 47, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.464894344Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"downstairs","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"[1]R generation[0..12]: [15, 29, 28, 21, 25, 27, 27, 27, 24, 24, 24, 28]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.464913042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"downstairs","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.464932659Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"downstairs","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"[2]R flush_numbers[0..12]: [29, 61, 57, 39, 48, 51, 51, 51, 47, 47, 47, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.464951157Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"downstairs","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"[2]R generation[0..12]: [15, 29, 28, 21, 25, 27, 27, 27, 24, 24, 24, 28]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.464969535Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"downstairs","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.464988992Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"downstairs","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"Max found gen is 30","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.46500534Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"downstairs","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"Generation requested: 30 >= found:30","v":0,"name":"crucible","Wait for a query_work_queue command to finish before sending IO level":30,"time":"2025-11-06T20:11:54.465028227Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"downstairs","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"Next flush: 62","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.465044405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"downstairs","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.465060773Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"":"downstairs","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.465078511Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.465093709Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"client":"0","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.465113077Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"client":"1","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.465130074Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"client":"2","":"downstairs","session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.465147232Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.46516321Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"4883eb6e-0c14-40d1-b8c9-d7685b1d81f5 is now active with session: 0915cbf0-3c32-4597-a215-2db7a1be014f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.465180988Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"}Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.465199936Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320,"session_id":"0915cbf0-3c32-4597-a215-2db7a1be014f"} {"msg":"The guest has finished waiting for activation with:30","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.465215844Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320} {"msg":"Activated sub_volume 4883eb6e-0c14-40d1-b8c9-d7685b1d81f5","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:54.465229662Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 535 len 12288 data: 3 2 2 02/30 Flush 03/30 Write block 463 len 20480 data: 3 4 5 4 5 04/30 Read block 521 len 12288 05/30 Write block 588 len 12288 data: 9 9 8 06/30 Read block 63 len 36864 07/30 Write block 290 len 4096 data: 3 08/30 Read block 137 len 4096 09/30 Write block 536 len 12288 data: 3 3 2 10/30 Write block 64 len 32768 data: 5 6 5 5 6 5 4 4 11/30 Write block 579 len 16384 data: 2 2 2 3 12/30 Write block 345 len 8192 data: 2 2 13/30 Read block 334 len 28672 14/30 Read block 204 len 36864 15/30 Write block 312 len 40960 data: 4 5 5 3 3 3 3 3 4 4 16/30 Read block 213 len 40960 17/30 Write block 591 len 24576 data: 5 5 3 3 3 5 18/30 Read block 251 len 40960 19/30 Read block 92 len 20480 20/30 Read block 111 len 4096 21/30 Write block 111 len 40960 data: 3 3 4 4 4 5 4 4 5 5 22/30 Write block 405 len 36864 data: 3 2 2 4 5 6 6 5 5 23/30 Write block 304 len 36864 data: 2 3 3 5 5 6 6 6 5 24/30 Read block 474 len 24576 25/30 Read block 385 len 16384 26/30 Read block 378 len 12288 27/30 Read block 240 len 32768 28/30 Read block 335 len 8192 29/30 Read block 101 len 8192 30/30 Write block 589 len 12288 data: 10 9 6 ---------------------------------------------------------------- Crucible gen:30 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Sent Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 5 Done Sent Done false 1003 Acked Read 3 Done Sent Done false 1004 Acked Write 3 Done Sent Done false 1005 Acked Read 9 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Read 1 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Write 8 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Write 2 Done Sent Done false 1012 Acked Read 7 Done Sent Done false 1013 Acked Read 9 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Write 6 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Read 5 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Write 10 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Write 9 Done Sent Done false 1023 Acked Read 6 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Read 3 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 1 32 Done 29 0 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) None Some(JobId(1001)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:54.484006696Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1320} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 29 29 29 61 61 61 F F F 2 040-059 28 28 28 57 57 57 F F F 3 060-079 21 21 21 39 39 39 T F T <--- 4 080-099 25 25 25 48 48 48 F F F 5 100-119 27 27 27 51 51 51 T F T <--- 6 120-139 27 27 27 51 51 51 T F T <--- 7 140-159 27 27 27 51 51 51 F F F 8 160-179 24 24 24 47 47 47 F F F 9 180-199 24 24 24 47 47 47 F F F 10 200-219 24 24 24 47 47 47 F F F 11 220-239 28 28 28 54 54 54 F F F 12 240-259 25 25 25 48 48 48 F F F 13 260-279 28 28 28 53 53 53 F F F 14 280-299 28 28 28 53 53 53 T F T <--- 15 300-319 28 28 28 60 60 60 T F T <--- 16 320-339 27 27 27 51 51 51 T F T <--- 17 340-359 28 28 28 59 59 59 T F T <--- 18 360-379 28 28 28 57 57 57 F F F 19 380-399 28 28 28 59 59 59 F F F 20 400-419 23 23 23 41 41 41 T F T <--- 21 420-439 29 29 29 61 61 61 F F F 22 440-459 27 27 27 51 51 51 F F F 23 460-479 29 29 29 61 61 61 T F T <--- 24 480-499 16 16 16 30 30 30 F F F 25 500-519 20 20 20 37 37 37 F F F 26 520-539 30 20 30 62 37 62 T F T <--- 27 540-559 28 28 28 58 58 58 F F F 28 560-579 4 4 4 8 8 8 T F T <--- 29 580-599 22 22 22 40 40 40 T F T <--- Max gen: 30, Max flush: 62 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 15 Verifying data now No change after verify Loop: 15 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 29 29 29 61 61 61 F F F 2 040-059 28 28 28 57 57 57 F F F 3 060-079 31 31 31 63 63 63 F F F 4 080-099 25 25 25 48 48 48 F F F 5 100-119 31 31 31 63 63 63 F F F 6 120-139 31 31 31 63 63 63 F F F 7 140-159 27 27 27 51 51 51 F F F 8 160-179 24 24 24 47 47 47 F F F 9 180-199 24 24 24 47 47 47 F F F 10 200-219 24 24 24 47 47 47 F F F 11 220-239 28 28 28 54 54 54 F F F 12 240-259 25 25 25 48 48 48 F F F 13 260-279 28 28 28 53 53 53 F F F 14 280-299 31 31 31 63 63 63 F F F 15 300-319 31 31 31 63 63 63 F F F 16 320-339 31 31 31 63 63 63 F F F 17 340-359 31 31 31 63 63 63 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 28 28 28 59 59 59 F F F 20 400-419 31 31 31 63 63 63 F F F 21 420-439 29 29 29 61 61 61 F F F 22 440-459 27 27 27 51 51 51 F F F 23 460-479 31 31 31 63 63 63 F F F 24 480-499 16 16 16 30 30 30 F F F 25 500-519 20 20 20 37 37 37 F F F 26 520-539 31 31 31 63 63 63 F F F 27 540-559 28 28 28 58 58 58 F F F 28 560-579 31 31 31 63 63 63 F F F 29 580-599 31 31 31 63 63 63 F F F Max gen: 31, Max flush: 63 Begin loop 16 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.892451745Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.938235547Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.93884078Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.938893474Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.938909851Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.938932159Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328} {"msg":"Crucible bda0e544-25d3-4c9f-b47d-f9c90a4e0c09 has session id: dc5dc922-e08d-46e3-b771-8deae2cb14a3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.939464362Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"Upstairs opts: Upstairs UUID: bda0e544-25d3-4c9f-b47d-f9c90a4e0c09, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.939523794Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"Crucible stats registered with UUID: bda0e544-25d3-4c9f-b47d-f9c90a4e0c09","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.939546671Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:58.939563079Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.939767334Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.939811058Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.939837985Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.940065696Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"task":"crutest"} {"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.940100941Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328} {"msg":"bda0e544-25d3-4c9f-b47d-f9c90a4e0c09 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.940141076Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.940162674Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"client":"0","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.940182571Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"client":"1","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.940212027Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"client":"2","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.940241274Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.94027021Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.940299707Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"ds_connection connected from Ok(127.0.0.1:34535)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.940419931Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"io task","client":"2","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"ds_connection connected from Ok(127.0.0.1:48682)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.940470795Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"io task","client":"1","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"ds_connection connected from Ok(127.0.0.1:38726)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.940557494Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"io task","client":"0","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.941752153Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"client":"1","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.941784969Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"client":"2","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.941816196Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"client":"0","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942329251Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942419569Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"[0]R flush_numbers[0..12]: [29, 61, 57, 63, 48, 63, 63, 51, 47, 47, 47, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942522587Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"downstairs","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"[0]R generation[0..12]: [15, 29, 28, 31, 25, 31, 31, 27, 24, 24, 24, 28]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942545644Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"downstairs","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942565731Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"downstairs","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"[1]R flush_numbers[0..12]: [29, 61, 57, 63, 48, 63, 63, 51, 47, 47, 47, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942584629Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"downstairs","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"[1]R generation[0..12]: [15, 29, 28, 31, 25, 31, 31, 27, 24, 24, 24, 28]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942603566Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"downstairs","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942623154Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"downstairs","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"[2]R flush_numbers[0..12]: [29, 61, 57, 63, 48, 63, 63, 51, 47, 47, 47, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942641592Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"downstairs","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"[2]R generation[0..12]: [15, 29, 28, 31, 25, 31, 31, 27, 24, 24, 24, 28]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942662929Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"downstairs","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942682796Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"downstairs","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942699224Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"downstairs","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"Generation requested: 32 >= found:32","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942715952Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"Next flush: 64","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942738819Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"downstairs","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942754907Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"":"downstairs","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942772175Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942787573Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"client":"0","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942804521Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"client":"1","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942821439Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"client":"2","":"downstairs","session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942838667Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942854674Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"bda0e544-25d3-4c9f-b47d-f9c90a4e0c09 is now active with session: dc5dc922-e08d-46e3-b771-8deae2cb14a3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942872642Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.9428879Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328,"session_id":"dc5dc922-e08d-46e3-b771-8deae2cb14a3"} {"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942904828Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328} {"msg":"Activated sub_volume bda0e544-25d3-4c9f-b47d-f9c90a4e0c09","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:11:58.942918786Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 129 len 20480 data: 6 6 5 5 5 02/30 Read block 586 len 28672 03/30 Read block 393 len 40960 04/30 Read block 424 len 36864 05/30 Write block 35 len 32768 data: 3 5 5 6 6 6 5 5 06/30 Read block 11 len 24576 07/30 Read block 447 len 12288 08/30 Write block 136 len 36864 data: 6 5 5 4 4 4 5 4 5 09/30 Write block 388 len 24576 data: 4 2 2 2 3 2 10/30 Read block 253 len 36864 11/30 Write block 401 len 16384 data: 4 3 2 3 12/30 Read block 424 len 28672 13/30 Read block 18 len 8192 14/30 Write block 533 len 16384 data: 5 4 4 4 15/30 Read block 194 len 4096 16/30 Read block 390 len 4096 17/30 Read block 462 len 4096 18/30 Read block 319 len 28672 19/30 Read block 510 len 36864 20/30 Read block 346 len 20480 21/30 Flush 22/30 Write block 395 len 36864 data: 3 3 4 3 3 3 5 4 3 23/30 Write block 159 len 20480 data: 4 3 3 3 2 24/30 Read block 298 len 8192 25/30 Write block 517 len 4096 data: 5 26/30 Write block 389 len 40960 data: 3 3 3 4 3 3 4 4 5 4 27/30 Write block 488 len 4096 data: 2 28/30 Write block 521 len 40960 data: 5 3 3 4 5 4 5 6 4 4 29/30 Read block 220 len 40960 30/30 Write block 188 len 16384 data: 4 4 3 3 ---------------------------------------------------------------- Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Read 9 Done Done Done false 1004 Acked Write 8 Done Done Done false 1005 Acked Read 6 Done Done Done false 1006 Acked Read 3 Done Done Done false 1007 Acked Write 9 Done Done Done false 1008 Acked Write 6 Done Done Done false 1009 Acked Read 9 Done Done Done false 1010 Acked Write 4 Done Done Done false 1011 Acked Read 7 Done Done Done false 1012 Acked Read 2 Done Done Done false 1013 Acked Write 4 Done Done Done false 1014 Acked Read 1 Done Done Done false 1015 Acked Read 1 Done Done Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Read 9 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Write 5 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Write 1 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 14 1 16 Done 29 16 29 74 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) None Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:11:58.962443547Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1328} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 32 29 32 64 61 64 F T F <--- 2 040-059 32 28 32 64 57 64 F T F <--- 3 060-079 31 31 31 63 63 63 F F F 4 080-099 25 25 25 48 48 48 F F F 5 100-119 31 31 31 63 63 63 F F F 6 120-139 32 31 32 64 63 64 F T F <--- 7 140-159 32 27 32 64 51 64 T T T <--- 8 160-179 24 24 24 47 47 47 T F T <--- 9 180-199 24 24 24 47 47 47 T F T <--- 10 200-219 24 24 24 47 47 47 F F F 11 220-239 28 28 28 54 54 54 F F F 12 240-259 25 25 25 48 48 48 F F F 13 260-279 28 28 28 53 53 53 F F F 14 280-299 31 31 31 63 63 63 F F F 15 300-319 31 31 31 63 63 63 F F F 16 320-339 31 31 31 63 63 63 F F F 17 340-359 31 31 31 63 63 63 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 32 28 32 64 59 64 T T T <--- 20 400-419 32 31 32 64 63 64 T T T <--- 21 420-439 29 29 29 61 61 61 F F F 22 440-459 27 27 27 51 51 51 F F F 23 460-479 31 31 31 63 63 63 F F F 24 480-499 16 16 16 30 30 30 T F T <--- 25 500-519 20 20 20 37 37 37 T F T <--- 26 520-539 32 31 32 64 63 64 T T T <--- 27 540-559 28 28 28 58 58 58 F F F 28 560-579 31 31 31 63 63 63 F F F 29 580-599 31 31 31 63 63 63 F F F Max gen: 32, Max flush: 64 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 16 Verifying data now No change after verify Loop: 16 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 32 32 32 64 64 64 F F F 2 040-059 32 32 32 64 64 64 F F F 3 060-079 31 31 31 63 63 63 F F F 4 080-099 25 25 25 48 48 48 F F F 5 100-119 31 31 31 63 63 63 F F F 6 120-139 32 32 32 64 64 64 F F F 7 140-159 33 33 33 65 65 65 F F F 8 160-179 33 33 33 65 65 65 F F F 9 180-199 33 33 33 65 65 65 F F F 10 200-219 24 24 24 47 47 47 F F F 11 220-239 28 28 28 54 54 54 F F F 12 240-259 25 25 25 48 48 48 F F F 13 260-279 28 28 28 53 53 53 F F F 14 280-299 31 31 31 63 63 63 F F F 15 300-319 31 31 31 63 63 63 F F F 16 320-339 31 31 31 63 63 63 F F F 17 340-359 31 31 31 63 63 63 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 33 33 33 65 65 65 F F F 20 400-419 33 33 33 65 65 65 F F F 21 420-439 29 29 29 61 61 61 F F F 22 440-459 27 27 27 51 51 51 F F F 23 460-479 31 31 31 63 63 63 F F F 24 480-499 33 33 33 65 65 65 F F F 25 500-519 33 33 33 65 65 65 F F F 26 520-539 33 33 33 65 65 65 F F F 27 540-559 28 28 28 58 58 58 F F F 28 560-579 31 31 31 63 63 63 F F F 29 580-599 31 31 31 63 63 63 F F F Max gen: 33, Max flush: 65 Begin loop 17 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.369738809Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:13.414546673Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.414947803Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.447490483Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.448046333Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.448094787Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.448111405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.448133272Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336} {"msg":"Crucible 2754c14b-6e6f-46d1-92e2-1a4abb8f07de has session id: 3a87e64f-864d-4283-a2f0-f81b36c7c9ec","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.448633779Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"Upstairs opts: Upstairs UUID: 2754c14b-6e6f-46d1-92e2-1a4abb8f07de, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.448690432Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"Crucible stats registered with UUID: 2754c14b-6e6f-46d1-92e2-1a4abb8f07de","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.448715609Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:13.448732946Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.448922033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.448959578Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.448978675Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.449123137Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"task":"crutest"} {"msg":"The guest has requested activation with gen:34","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.449148544Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336} {"msg":"2754c14b-6e6f-46d1-92e2-1a4abb8f07de active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.449212736Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.449232283Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"client":"0","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.449252801Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"client":"1","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.449349609Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"client":"2","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.449379855Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.44942087Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.449461405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:54175)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.449493051Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"io task","client":"2","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:43533)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.449524767Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"io task","client":"0","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"ds_connection connected from Ok(127.0.0.1:50599)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.449559152Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"io task","client":"1","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.450727245Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"client":"1","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.45077218Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"client":"0","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.450799596Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"client":"2","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451307402Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.45140386Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"[0]R flush_numbers[0..12]: [29, 64, 64, 63, 48, 63, 64, 65, 65, 65, 47, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451490649Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"downstairs","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"[0]R generation[0..12]: [15, 32, 32, 31, 25, 31, 32, 33, 33, 33, 24, 28]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451515096Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"downstairs","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451535433Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"downstairs","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"[1]R flush_numbers[0..12]: [29, 64, 64, 63, 48, 63, 64, 65, 65, 65, 47, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451554191Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"downstairs","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"[1]R generation[0..12]: [15, 32, 32, 31, 25, 31, 32, 33, 33, 33, 24, 28]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451572749Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"downstairs","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451592046Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"downstairs","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"[2]R flush_numbers[0..12]: [29, 64, 64, 63, 48, 63, 64, 65, 65, 65, 47, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451610534Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"downstairs","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"[2]R generation[0..12]: [15, 32, 32, 31, 25, 31, 32, 33, 33, 33, 24, 28]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451628912Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"downstairs","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451648449Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"downstairs","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"Max found gen is 34","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451664507Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"downstairs","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"Generation requested: 34 >= found:34","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-11-06T20:12:13.451681145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"downstairs","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"Next flush: 66","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451702482Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"downstairs","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.45171834Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"":"downstairs","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451738628Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451754216Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"client":"0","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451771054Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"client":"1","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451787572Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"client":"2","":"downstairs","session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.45180463Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451820677Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"2754c14b-6e6f-46d1-92e2-1a4abb8f07de is now active with session: 3a87e64f-864d-4283-a2f0-f81b36c7c9ec","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451838735Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451853803Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336,"session_id":"3a87e64f-864d-4283-a2f0-f81b36c7c9ec"} {"msg":"The guest has finished waiting for activation with:34","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.451869711Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336} {"msg":"Activated sub_volume 2754c14b-6e6f-46d1-92e2-1a4abb8f07de","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:13.45188386Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 238 len 32768 data: 2 3 2 3 3 4 4 4 03/30 Write block 129 len 36864 data: 7 7 6 6 6 5 5 7 6 04/30 Read block 256 len 32768 05/30 Read block 489 len 24576 06/30 Read block 59 len 4096 07/30 Read block 544 len 12288 08/30 Write block 297 len 36864 data: 2 2 2 2 2 2 2 3 4 09/30 Read block 426 len 28672 10/30 Read block 524 len 12288 11/30 Flush 12/30 Write block 233 len 36864 data: 3 2 3 3 2 3 4 3 4 13/30 Read block 569 len 32768 14/30 Read block 516 len 36864 15/30 Read block 455 len 20480 16/30 Flush 17/30 Write block 123 len 28672 data: 4 4 4 4 5 6 8 18/30 Read block 523 len 16384 19/30 Write block 506 len 32768 data: 2 2 3 3 3 3 3 3 20/30 Read block 255 len 20480 21/30 Read block 408 len 16384 22/30 Write block 270 len 12288 data: 4 4 4 23/30 Write block 341 len 32768 data: 4 3 3 3 3 3 2 2 24/30 Read block 446 len 12288 25/30 Write block 58 len 4096 data: 3 26/30 Write block 20 len 36864 data: 3 3 4 4 4 6 5 4 3 27/30 Read block 99 len 4096 28/30 Read block 58 len 4096 29/30 Read block 403 len 40960 30/30 Read block 544 len 8192 ---------------------------------------------------------------- Crucible gen:34 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Sent Done Done false 1001 Acked Write 8 Sent Done Done false 1002 Acked Write 9 Sent Done Done false 1003 Acked Read 8 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Read 1 Sent Done Done false 1006 Acked Read 3 Sent Done Done false 1007 Acked Write 9 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Read 3 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 9 Sent Done Done false 1012 Acked Read 8 Sent Done Done false 1013 Acked Read 9 Sent Done Done false 1014 Acked Read 5 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Read 4 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Read 4 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Write 1 Sent Done Done false 1025 Acked Write 9 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Read 2 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 0 31 Done 0 29 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:13.473243638Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1336} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 32 32 32 64 64 64 F T T <--- 2 040-059 32 32 32 64 64 64 F T T <--- 3 060-079 31 31 31 63 63 63 F F F 4 080-099 25 25 25 48 48 48 F F F 5 100-119 31 31 31 63 63 63 F F F 6 120-139 32 34 34 64 67 67 F T T <--- 7 140-159 33 33 33 65 65 65 F F F 8 160-179 33 33 33 65 65 65 F F F 9 180-199 33 33 33 65 65 65 F F F 10 200-219 24 24 24 47 47 47 F F F 11 220-239 28 34 34 54 68 68 F F F <--- 12 240-259 25 34 34 48 68 68 F F F <--- 13 260-279 28 28 28 53 53 53 F T T <--- 14 280-299 31 34 34 63 67 67 F F F <--- 15 300-319 31 34 34 63 67 67 F F F <--- 16 320-339 31 31 31 63 63 63 F F F 17 340-359 31 31 31 63 63 63 F T T <--- 18 360-379 28 28 28 57 57 57 F F F 19 380-399 33 33 33 65 65 65 F F F 20 400-419 33 33 33 65 65 65 F F F 21 420-439 29 29 29 61 61 61 F F F 22 440-459 27 27 27 51 51 51 F F F 23 460-479 31 31 31 63 63 63 F F F 24 480-499 33 33 33 65 65 65 F F F 25 500-519 33 33 33 65 65 65 F T T <--- 26 520-539 33 33 33 65 65 65 F F F 27 540-559 28 28 28 58 58 58 F F F 28 560-579 31 31 31 63 63 63 F F F 29 580-599 31 31 31 63 63 63 F F F Max gen: 34, Max flush: 68 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 17 Verifying data now No change after verify Loop: 17 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 F F F 1 020-039 35 35 35 69 69 69 F F F 2 040-059 35 35 35 69 69 69 F F F 3 060-079 31 31 31 63 63 63 F F F 4 080-099 25 25 25 48 48 48 F F F 5 100-119 31 31 31 63 63 63 F F F 6 120-139 35 35 35 69 69 69 F F F 7 140-159 33 33 33 65 65 65 F F F 8 160-179 33 33 33 65 65 65 F F F 9 180-199 33 33 33 65 65 65 F F F 10 200-219 24 24 24 47 47 47 F F F 11 220-239 34 34 34 68 68 68 F F F 12 240-259 34 34 34 68 68 68 F F F 13 260-279 35 35 35 69 69 69 F F F 14 280-299 34 34 34 67 67 67 F F F 15 300-319 34 34 34 67 67 67 F F F 16 320-339 31 31 31 63 63 63 F F F 17 340-359 35 35 35 69 69 69 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 33 33 33 65 65 65 F F F 20 400-419 33 33 33 65 65 65 F F F 21 420-439 29 29 29 61 61 61 F F F 22 440-459 27 27 27 51 51 51 F F F 23 460-479 31 31 31 63 63 63 F F F 24 480-499 33 33 33 65 65 65 F F F 25 500-519 35 35 35 69 69 69 F F F 26 520-539 33 33 33 65 65 65 F F F 27 540-559 28 28 28 58 58 58 F F F 28 560-579 31 31 31 63 63 63 F F F 29 580-599 31 31 31 63 63 63 F F F Max gen: 35, Max flush: 69 Begin loop 18 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.895864821Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.941972642Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.942637148Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.942684562Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.94270088Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.942729127Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344} {"msg":"Crucible 145400f0-cc23-4ec4-9d83-166d062c2dcc has session id: a689cb2e-f25e-43a4-aada-c78e614351f1","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.943403972Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"Upstairs opts: Upstairs UUID: 145400f0-cc23-4ec4-9d83-166d062c2dcc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.943459744Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"Crucible stats registered with UUID: 145400f0-cc23-4ec4-9d83-166d062c2dcc","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.943490371Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:17.943515948Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.943767386Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.94381096Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.943832328Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.943979139Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"task":"crutest"} {"msg":"The guest has requested activation with gen:36","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.943999896Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344} {"msg":"145400f0-cc23-4ec4-9d83-166d062c2dcc active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.944087065Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.944107863Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"client":"0","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.944125351Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"client":"1","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.944142269Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"client":"2","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.944158816Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.944176304Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.944236767Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"ds_connection connected from Ok(127.0.0.1:42532)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.94428582Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"io task","client":"0","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"ds_connection connected from Ok(127.0.0.1:62913)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.944327365Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"io task","client":"2","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"ds_connection connected from Ok(127.0.0.1:49062)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.944363351Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"io task","client":"1","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.945678065Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"client":"1","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.945712531Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"client":"0","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.945734648Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"client":"2","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.94619112Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946281819Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"[0]R flush_numbers[0..12]: [29, 69, 69, 63, 48, 63, 69, 65, 65, 65, 47, 68]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946312685Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"downstairs","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"[0]R generation[0..12]: [15, 35, 35, 31, 25, 31, 35, 33, 33, 33, 24, 34]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946332513Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"downstairs","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.94635233Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"downstairs","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"[1]R flush_numbers[0..12]: [29, 69, 69, 63, 48, 63, 69, 65, 65, 65, 47, 68]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946371378Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"downstairs","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"[1]R generation[0..12]: [15, 35, 35, 31, 25, 31, 35, 33, 33, 33, 24, 34]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946391005Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"downstairs","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946410803Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"downstairs","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"[2]R flush_numbers[0..12]: [29, 69, 69, 63, 48, 63, 69, 65, 65, 65, 47, 68]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.94643389Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"downstairs","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"[2]R generation[0..12]: [15, 35, 35, 31, 25, 31, 35, 33, 33, 33, 24, 34]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946452938Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"downstairs","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946472405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"downstairs","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"Max found gen is 36","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946488473Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"downstairs","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"Generation requested: 36 >= found:36","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946505011Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"downstairs","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"Next flush: 70","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946521009Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"downstairs","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946536947Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"":"downstairs","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-11-06T20:12:17.946553955Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946574272Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"client":"0","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.94659094Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"client":"1","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946607818Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"client":"2","":"downstairs","session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946624956Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946641164Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"145400f0-cc23-4ec4-9d83-166d062c2dcc is now active with session: a689cb2e-f25e-43a4-aada-c78e614351f1","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946659081Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.94667388Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344,"session_id":"a689cb2e-f25e-43a4-aada-c78e614351f1"} {"msg":"The guest has finished waiting for activation with:36","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946692197Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344} {"msg":"Activated sub_volume 145400f0-cc23-4ec4-9d83-166d062c2dcc","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:17.946706586Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 277 len 8192 data: 5 5 02/30 Read block 457 len 20480 03/30 Read block 332 len 36864 04/30 Write block 140 len 28672 data: 5 5 6 5 6 5 2 05/30 Write block 533 len 40960 data: 6 5 5 5 4 3 2 3 4 6 06/30 Read block 399 len 40960 07/30 Write block 116 len 12288 data: 6 5 5 08/30 Write block 254 len 24576 data: 4 4 4 4 5 2 09/30 Read block 41 len 8192 10/30 Write block 176 len 32768 data: 4 4 4 7 7 7 7 7 11/30 Read block 492 len 4096 12/30 Write block 229 len 40960 data: 4 4 5 3 4 3 4 4 3 4 13/30 Read block 540 len 24576 14/30 Read block 553 len 32768 15/30 Read block 153 len 12288 16/30 Read block 25 len 16384 17/30 Read block 230 len 4096 18/30 Read block 311 len 24576 19/30 Write block 3 len 40960 data: 2 2 2 2 2 2 2 2 3 3 20/30 Read block 10 len 32768 21/30 Read block 345 len 36864 22/30 Read block 512 len 16384 23/30 Read block 146 len 28672 24/30 Read block 516 len 28672 25/30 Read block 187 len 12288 26/30 Read block 522 len 32768 27/30 Read block 8 len 28672 28/30 Write block 545 len 16384 data: 6 6 6 7 29/30 Read block 50 len 8192 30/30 Write block 321 len 32768 data: 5 4 5 5 6 6 5 4 ---------------------------------------------------------------- Crucible gen:36 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 5 Done Done Done false 1002 Acked Read 9 Done Done Done false 1003 Acked Write 7 Done Done Sent false 1004 Acked Write 10 Done Done Sent false 1005 Acked Read 10 Done Done Sent false 1006 Acked Write 3 Done Done Sent false 1007 Acked Write 6 Done Done Sent false 1008 Acked Read 2 Done Done Sent false 1009 Acked Write 8 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Read 6 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Read 3 Done Done Sent false 1015 Acked Read 4 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Write 10 Done Done Sent false 1019 Acked Read 8 Done Done Sent false 1020 Acked Read 9 Done Done Sent false 1021 Acked Read 4 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Read 3 Done Done Sent false 1025 Acked Read 8 Done Done Sent false 1026 Acked Read 7 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Read 2 Done Done Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 27 29 Done 29 29 3 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:17.966706486Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1344} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 15 15 15 29 29 29 T T F <--- 1 020-039 35 35 35 69 69 69 F F F 2 040-059 35 35 35 69 69 69 F F F 3 060-079 31 31 31 63 63 63 F F F 4 080-099 25 25 25 48 48 48 F F F 5 100-119 31 31 31 63 63 63 T T F <--- 6 120-139 35 35 35 69 69 69 F F F 7 140-159 33 33 33 65 65 65 T T F <--- 8 160-179 33 33 33 65 65 65 T T F <--- 9 180-199 33 33 33 65 65 65 T T F <--- 10 200-219 24 24 24 47 47 47 F F F 11 220-239 34 34 34 68 68 68 T T F <--- 12 240-259 34 34 34 68 68 68 T T F <--- 13 260-279 35 35 35 69 69 69 T T T 14 280-299 34 34 34 67 67 67 F F F 15 300-319 34 34 34 67 67 67 F F F 16 320-339 31 31 31 63 63 63 T T F <--- 17 340-359 35 35 35 69 69 69 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 33 33 33 65 65 65 F F F 20 400-419 33 33 33 65 65 65 F F F 21 420-439 29 29 29 61 61 61 F F F 22 440-459 27 27 27 51 51 51 F F F 23 460-479 31 31 31 63 63 63 F F F 24 480-499 33 33 33 65 65 65 F F F 25 500-519 35 35 35 69 69 69 F F F 26 520-539 33 33 33 65 65 65 T T F <--- 27 540-559 28 28 28 58 58 58 T T F <--- 28 560-579 31 31 31 63 63 63 F F F 29 580-599 31 31 31 63 63 63 F F F Max gen: 35, Max flush: 69 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 18 Verifying data now No change after verify Loop: 18 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 70 70 70 F F F 1 020-039 35 35 35 69 69 69 F F F 2 040-059 35 35 35 69 69 69 F F F 3 060-079 31 31 31 63 63 63 F F F 4 080-099 25 25 25 48 48 48 F F F 5 100-119 36 36 36 70 70 70 F F F 6 120-139 35 35 35 69 69 69 F F F 7 140-159 36 36 36 70 70 70 F F F 8 160-179 36 36 36 70 70 70 F F F 9 180-199 36 36 36 70 70 70 F F F 10 200-219 24 24 24 47 47 47 F F F 11 220-239 36 36 36 70 70 70 F F F 12 240-259 36 36 36 70 70 70 F F F 13 260-279 36 36 36 70 70 70 F F F 14 280-299 34 34 34 67 67 67 F F F 15 300-319 34 34 34 67 67 67 F F F 16 320-339 36 36 36 70 70 70 F F F 17 340-359 35 35 35 69 69 69 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 33 33 33 65 65 65 F F F 20 400-419 33 33 33 65 65 65 F F F 21 420-439 29 29 29 61 61 61 F F F 22 440-459 27 27 27 51 51 51 F F F 23 460-479 31 31 31 63 63 63 F F F 24 480-499 33 33 33 65 65 65 F F F 25 500-519 35 35 35 69 69 69 F F F 26 520-539 36 36 36 70 70 70 F F F 27 540-559 36 36 36 70 70 70 F F F 28 560-579 31 31 31 63 63 63 F F F 29 580-599 31 31 31 63 63 63 F F F Max gen: 36, Max flush: 70 Begin loop 19 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.366171606Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.412528216Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.413149937Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.413198001Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.413214319Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.413236457Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352} {"msg":"Crucible 5c60b640-2373-4acb-aad0-a0c75d4ff7ec has session id: 970e9a55-492f-43df-bddd-2e6d4376910e","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.413913841Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"Upstairs opts: Upstairs UUID: 5c60b640-2373-4acb-aad0-a0c75d4ff7ec, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.413960585Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"Crucible stats registered with UUID: 5c60b640-2373-4acb-aad0-a0c75d4ff7ec","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.413981743Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:22.41399827Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414224702Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"io task","client":"2","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414263267Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"io task","client":"1","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414286684Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"io task","client":"0","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414413648Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"task":"crutest"} {"msg":"The guest has requested activation with gen:38","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414465871Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352} {"msg":"5c60b640-2373-4acb-aad0-a0c75d4ff7ec active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414494808Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414522924Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"client":"0","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414586126Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"client":"1","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414612773Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"client":"2","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414629961Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"io task","client":"2","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414650918Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"io task","client":"0","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414681964Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"io task","client":"1","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"ds_connection connected from Ok(127.0.0.1:55642)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.41471821Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"io task","client":"2","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"ds_connection connected from Ok(127.0.0.1:59164)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414751895Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"io task","client":"0","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"ds_connection connected from Ok(127.0.0.1:54625)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.414893038Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"io task","client":"1","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.415949615Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"client":"0","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416016856Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"client":"1","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416059481Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"client":"2","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416524282Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416616611Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"[0]R flush_numbers[0..12]: [70, 69, 69, 63, 48, 70, 69, 70, 70, 70, 47, 70]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416646587Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"downstairs","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"[0]R generation[0..12]: [36, 35, 35, 31, 25, 36, 35, 36, 36, 36, 24, 36]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416666484Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"downstairs","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416687282Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"downstairs","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"[1]R flush_numbers[0..12]: [70, 69, 69, 63, 48, 70, 69, 70, 70, 70, 47, 70]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416705889Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"downstairs","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"[1]R generation[0..12]: [36, 35, 35, 31, 25, 36, 35, 36, 36, 36, 24, 36]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416724217Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"downstairs","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416743755Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"downstairs","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"[2]R flush_numbers[0..12]: [70, 69, 69, 63, 48, 70, 69, 70, 70, 70, 47, 70]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416762592Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"downstairs","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"[2]R generation[0..12]: [36, 35, 35, 31, 25, 36, 35, 36, 36, 36, 24, 36]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.41678214Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"downstairs","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416815146Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"downstairs","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"Max found gen is 37","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416846102Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"downstairs","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"Wait for a query_work_queue command to finish before sending IO Generation requested: 38 >= found:37","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416874068Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"downstairs","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"Next flush: 71","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416890636Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"downstairs","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416906544Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"":"downstairs","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416927232Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.41694271Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"client":"0","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416959637Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"client":"1","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416976265Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"client":"2","":"downstairs","session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.416993273Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.417011551Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"5c60b640-2373-4acb-aad0-a0c75d4ff7ec is now active with session: 970e9a55-492f-43df-bddd-2e6d4376910e","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.417029508Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.417044657Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352,"session_id":"970e9a55-492f-43df-bddd-2e6d4376910e"} {"msg":"The guest has finished waiting for activation with:38","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:22.417060685Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352} {"msg":"Activated sub_volume 5c60b640-2373-4acb-aad0-a0c75d4ff7ec","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:12:22.417074783Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 128 len 32768 data: 7 9 8 7 7 7 6 6 02/30 Write block 501 len 12288 data: 4 4 4 03/30 Read block 57 len 4096 04/30 Write block 525 len 20480 data: 6 5 6 7 5 05/30 Read block 569 len 32768 06/30 Read block 148 len 8192 07/30 Write block 492 len 8192 data: 2 2 08/30 Read block 282 len 20480 09/30 Flush 10/30 Write block 387 len 28672 data: 4 5 4 4 4 5 4 11/30 Write block 72 len 40960 data: 4 2 2 2 2 2 2 3 3 3 12/30 Read block 256 len 40960 13/30 Write block 567 len 4096 data: 3 14/30 Write block 439 len 4096 data: 5 15/30 Read block 536 len 20480 16/30 Read block 436 len 16384 17/30 Read block 307 len 28672 18/30 Write block 76 len 36864 data: 3 3 3 4 4 4 4 4 4 19/30 Write block 550 len 20480 data: 5 4 4 5 7 20/30 Read block 548 len 40960 21/30 Read block 503 len 8192 22/30 Read block 16 len 24576 23/30 Read block 580 len 16384 24/30 Write block 99 len 4096 data: 3 25/30 Flush 26/30 Write block 138 len 28672 data: 6 5 6 6 7 6 7 27/30 Write block 475 len 4096 data: 5 28/30 Write block 82 len 8192 data: 5 5 29/30 Read block 422 len 40960 30/30 Write block 242 len 20480 data: 4 5 5 5 4 ---------------------------------------------------------------- Crucible gen:38 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 3 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Write 5 Done Sent Done false 1004 Acked Read 8 Done Sent Done false 1005 Acked Read 2 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 7 Done Sent Done false 1010 Acked Write 10 Done Sent Done false 1011 Acked Read 10 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Read 5 Done Sent Done false 1015 Acked Read 4 Done Sent Done false 1016 Acked Read 7 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Read 4 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Read 10 Done Sent Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 1 29 Done 29 3 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:22.435390245Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1352} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 70 70 70 F F F 1 020-039 35 35 35 69 69 69 F F F 2 040-059 35 35 35 69 69 69 F F F 3 060-079 38 31 38 72 63 72 F F F <--- 4 080-099 38 25 38 72 48 72 T F T <--- 5 100-119 36 36 36 70 70 70 F F F 6 120-139 38 35 38 71 69 71 T T T <--- 7 140-159 36 36 36 70 70 70 T F T <--- 8 160-179 36 36 36 70 70 70 F F F 9 180-199 36 36 36 70 70 70 F F F 10 200-219 24 24 24 47 47 47 F F F 11 220-239 36 36 36 70 70 70 F F F 12 240-259 36 36 36 70 70 70 T F T <--- 13 260-279 36 36 36 70 70 70 F F F 14 280-299 34 34 34 67 67 67 F F F 15 300-319 34 34 34 67 67 67 F F F 16 320-339 36 36 36 70 70 70 F F F 17 340-359 35 35 35 69 69 69 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 38 33 38 72 65 72 F F F <--- 20 400-419 33 33 33 65 65 65 F F F 21 420-439 38 29 38 72 61 72 F F F <--- 22 440-459 27 27 27 51 51 51 F F F 23 460-479 31 31 31 63 63 63 T F T <--- 24 480-499 38 33 38 71 65 71 F F F <--- 25 500-519 38 35 38 71 69 71 F T F <--- 26 520-539 38 36 38 71 70 71 F F F <--- 27 540-559 38 36 38 72 70 72 F F F <--- 28 560-579 38 31 38 72 63 72 F F F <--- 29 580-599 31 31 31 63 63 63 F F F Max gen: 38, Max flush: 72 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 19 Verifying data now No change after verify Loop: 19 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 70 70 70 F F F 1 020-039 35 35 35 69 69 69 F F F 2 040-059 35 35 35 69 69 69 F F F 3 060-079 38 38 38 72 72 72 F F F 4 080-099 39 39 39 73 73 73 F F F 5 100-119 36 36 36 70 70 70 F F F 6 120-139 39 39 39 73 73 73 F F F 7 140-159 39 39 39 73 73 73 F F F 8 160-179 36 36 36 70 70 70 F F F 9 180-199 36 36 36 70 70 70 F F F 10 200-219 24 24 24 47 47 47 F F F 11 220-239 36 36 36 70 70 70 F F F 12 240-259 39 39 39 73 73 73 F F F 13 260-279 36 36 36 70 70 70 F F F 14 280-299 34 34 34 67 67 67 F F F 15 300-319 34 34 34 67 67 67 F F F 16 320-339 36 36 36 70 70 70 F F F 17 340-359 35 35 35 69 69 69 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 38 38 38 72 72 72 F F F 20 400-419 33 33 33 65 65 65 F F F 21 420-439 38 38 38 72 72 72 F F F 22 440-459 27 27 27 51 51 51 F F F 23 460-479 39 39 39 73 73 73 F F F 24 480-499 38 38 38 71 71 71 F F F 25 500-519 38 38 38 71 71 71 F F F 26 520-539 38 38 38 71 71 71 F F F 27 540-559 38 38 38 72 72 72 F F F 28 560-579 38 38 38 72 72 72 F F F 29 580-599 31 31 31 63 63 63 F F F Max gen: 39, Max flush: 73 Begin loop 20 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.840948138Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:26.885426404Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.885820325Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.917791947Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.918430236Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.918488179Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.918507396Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.918539343Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360} {"msg":"Crucible e855c0c7-9ca7-4101-b748-308024b2fed4 has session id: 14cd5500-8ce3-460a-9b01-9f3c08c35a8d","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919134947Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"Upstairs opts: Upstairs UUID: e855c0c7-9ca7-4101-b748-308024b2fed4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919199229Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"Crucible stats registered with UUID: e855c0c7-9ca7-4101-b748-308024b2fed4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919222906Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:26.919240594Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919438429Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"io task","client":"2","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919475405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"io task","client":"1","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919494992Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"io task","client":"0","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919699896Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"task":"crutest"} {"msg":"The guest has requested activation with gen:40","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919723433Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360} {"msg":"e855c0c7-9ca7-4101-b748-308024b2fed4 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919800333Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919820611Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"client":"0","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919838109Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"client":"1","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919854957Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"client":"2","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919871655Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"io task","client":"2","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.919892512Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"io task","client":"0","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.91990993Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"io task","client":"1","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:46019)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.920157079Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"io task","client":"1","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:35306)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.920203203Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"io task","client":"2","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"ds_connection connected from Ok(127.0.0.1:63706)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.92022399Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"io task","client":"0","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.921353988Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"client":"2","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.921432558Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"client":"1","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.921634793Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"client":"0","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.921981149Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922059549Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"[0]R flush_numbers[0..12]: [70, 69, 69, 72, 73, 70, 73, 73, 70, 70, 47, 70]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922170235Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"downstairs","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"[0]R generation[0..12]: [36, 35, 35, 38, 39, 36, 39, 39, 36, 36, 24, 36]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922195482Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"downstairs","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922216349Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"downstairs","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"[1]R flush_numbers[0..12]: [70, 69, 69, 72, 73, 70, 73, 73, 70, 70, 47, 70]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922235037Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"downstairs","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"[1]R generation[0..12]: [36, 35, 35, 38, 39, 36, 39, 39, 36, 36, 24, 36]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922253364Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"downstairs","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922276981Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"downstairs","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"[2]R flush_numbers[0..12]: [70, 69, 69, 72, 73, 70, 73, 73, 70, 70, 47, 70]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922296209Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"downstairs","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"[2]R generation[0..12]: [36, 35, 35, 38, 39, 36, 39, 39, 36, 36, 24, 36]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922314827Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"downstairs","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922334684Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"downstairs","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"Max found gen is 40","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922351002Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"downstairs","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 40 >= found:40","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922373819Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"downstairs","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"Next flush: 74","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922390137Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"downstairs","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922406265Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"":"downstairs","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922423383Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922438681Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"client":"0","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922455899Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"client":"1","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922473027Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"client":"2","":"downstairs","session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922490265Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922506483Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"e855c0c7-9ca7-4101-b748-308024b2fed4 is now active with session: 14cd5500-8ce3-460a-9b01-9f3c08c35a8d","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.92252436Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922545188Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360,"session_id":"14cd5500-8ce3-460a-9b01-9f3c08c35a8d"} {"msg":"The guest has finished waiting for activation with:40","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922561696Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360} {"msg":"Activated sub_volume e855c0c7-9ca7-4101-b748-308024b2fed4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:26.922575814Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 296 len 40960 data: 3 3 3 3 3 3 3 3 4 5 02/30 Flush 03/30 Write block 331 len 36864 data: 3 4 4 4 3 3 3 4 5 04/30 Flush 05/30 Write block 23 len 24576 data: 5 5 7 6 5 4 06/30 Write block 455 len 8192 data: 6 6 07/30 Write block 444 len 36864 data: 3 3 2 2 3 3 4 5 5 08/30 Read block 142 len 36864 09/30 Flush 10/30 Write block 481 len 36864 data: 4 3 3 2 2 2 2 3 2 11/30 Write block 464 len 4096 data: 5 12/30 Read block 97 len 12288 13/30 Flush 14/30 Write block 201 len 20480 data: 4 4 5 6 6 15/30 Write block 109 len 36864 data: 3 3 4 4 5 5 5 7 6 16/30 Flush 17/30 Write block 434 len 8192 data: 6 5 18/30 Read block 263 len 24576 19/30 Read block 201 len 36864 20/30 Write block 232 len 16384 data: 4 5 4 5 21/30 Read block 285 len 36864 22/30 Write block 154 len 12288 data: 3 3 3 23/30 Read block 210 len 12288 24/30 Read block 64 len 36864 25/30 Write block 389 len 24576 data: 5 5 5 6 5 4 26/30 Read block 512 len 28672 27/30 Read block 103 len 12288 28/30 Read block 113 len 28672 29/30 Write block 306 len 8192 data: 4 6 30/30 Read block 223 len 16384 ---------------------------------------------------------------- Crucible gen:40 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Sent Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 9 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 6 Sent Done Done false 1005 Acked Write 2 Sent Done Done false 1006 Acked Write 9 Sent Done Done false 1007 Acked Read 9 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 9 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 2 Sent Done Done false 1017 Acked Read 6 Sent Done Done false 1018 Acked Read 9 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Read 9 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Read 3 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Write 2 Sent Done Done false 1029 Acked Read 4 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 1 31 Done 0 30 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:26.941952053Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1360} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 70 70 70 F F F 1 020-039 35 40 40 69 76 76 F F F <--- 2 040-059 35 35 35 69 69 69 F F F 3 060-079 38 38 38 72 72 72 F F F 4 080-099 39 39 39 73 73 73 F F F 5 100-119 36 40 40 70 78 78 F F F <--- 6 120-139 39 39 39 73 73 73 F F F 7 140-159 39 39 39 73 73 73 F T T <--- 8 160-179 36 36 36 70 70 70 F F F 9 180-199 36 36 36 70 70 70 F F F 10 200-219 24 40 40 47 78 78 F F F <--- 11 220-239 36 36 36 70 70 70 F T T <--- 12 240-259 39 39 39 73 73 73 F F F 13 260-279 36 36 36 70 70 70 F F F 14 280-299 34 40 40 67 74 74 F F F <--- 15 300-319 34 40 40 67 74 74 F T T <--- 16 320-339 36 40 40 70 75 75 F F F <--- 17 340-359 35 35 35 69 69 69 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 38 38 38 72 72 72 F T T <--- 20 400-419 33 33 33 65 65 65 F F F 21 420-439 38 38 38 72 72 72 F T T <--- 22 440-459 27 40 40 51 76 76 F F F <--- 23 460-479 39 40 40 73 77 77 F F F <--- 24 480-499 38 40 40 71 77 77 F F F <--- 25 500-519 38 38 38 71 71 71 F F F 26 520-539 38 38 38 71 71 71 F F F 27 540-559 38 38 38 72 72 72 F F F 28 560-579 38 38 38 72 72 72 F F F 29 580-599 31 31 31 63 63 63 F F F Max gen: 40, Max flush: 78 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 20 Verifying data now No change after verify Loop: 20 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 70 70 70 F F F 1 020-039 40 40 40 76 76 76 F F F 2 040-059 35 35 35 69 69 69 F F F 3 060-079 38 38 38 72 72 72 F F F 4 080-099 39 39 39 73 73 73 F F F 5 100-119 40 40 40 78 78 78 F F F 6 120-139 39 39 39 73 73 73 F F F 7 140-159 41 41 41 79 79 79 F F F 8 160-179 36 36 36 70 70 70 F F F 9 180-199 36 36 36 70 70 70 F F F 10 200-219 40 40 40 78 78 78 F F F 11 220-239 41 41 41 79 79 79 F F F 12 240-259 39 39 39 73 73 73 F F F 13 260-279 36 36 36 70 70 70 F F F 14 280-299 40 40 40 74 74 74 F F F 15 300-319 41 41 41 79 79 79 F F F 16 320-339 40 40 40 75 75 75 F F F 17 340-359 35 35 35 69 69 69 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 41 41 41 79 79 79 F F F 20 400-419 33 33 33 65 65 65 F F F 21 420-439 41 41 41 79 79 79 F F F 22 440-459 40 40 40 76 76 76 F F F 23 460-479 40 40 40 77 77 77 F F F 24 480-499 40 40 40 77 77 77 F F F 25 500-519 38 38 38 71 71 71 F F F 26 520-539 38 38 38 71 71 71 F F F 27 540-559 38 38 38 72 72 72 F F F 28 560-579 38 38 38 72 72 72 F F F 29 580-599 31 31 31 63 63 63 F F F Max gen: 41, Max flush: 79 Begin loop 21 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.383455437Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.429846802Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.430773075Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.430833128Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.430853185Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.430880831Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368} {"msg":"Crucible 8e8b8ea3-c28f-4e1d-86ed-cc93ddf3b28d has session id: 35a0a5af-cb3e-425c-8d7a-403220958dc0","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.431547347Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"Upstairs opts: Upstairs UUID: 8e8b8ea3-c28f-4e1d-86ed-cc93ddf3b28d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.431592052Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"Crucible stats registered with UUID: 8e8b8ea3-c28f-4e1d-86ed-cc93ddf3b28d","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.431613529Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:31.431629997Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.431859738Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"io task","client":"0","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.431911302Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"io task","client":"1","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.431948597Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"io task","client":"2","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.43216348Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"task":"crutest"} {"msg":"The guest has requested activation with gen:42","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.432197176Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368} {"msg":"8e8b8ea3-c28f-4e1d-86ed-cc93ddf3b28d active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.432259238Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.432280335Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"client":"0","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.432297883Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"client":"1","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.432315551Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"client":"2","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.432344227Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"io task","client":"2","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.432376553Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"io task","client":"0","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.432408929Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"io task","client":"1","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"ds_connection connected from Ok(127.0.0.1:55826)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.432443814Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"io task","client":"1","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"ds_connection connected from Ok(127.0.0.1:53756)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.432474321Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"io task","client":"0","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"ds_connection connected from Ok(127.0.0.1:55619)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.432507346Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"io task","client":"2","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.433846318Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"client":"1","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.433880844Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"client":"2","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.433951015Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"client":"0","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434553609Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434658315Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"[0]R flush_numbers[0..12]: [70, 76, 69, 72, 73, 78, 73, 79, 70, 70, 78, 79]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434693911Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"downstairs","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"[0]R generation[0..12]: [36, 40, 35, 38, 39, 40, 39, 41, 36, 36, 40, 41]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434713959Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"downstairs","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434733906Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"downstairs","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"[1]R flush_numbers[0..12]: [70, 76, 69, 72, 73, 78, 73, 79, 70, 70, 78, 79]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434752884Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"downstairs","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"[1]R generation[0..12]: [36, 40, 35, 38, 39, 40, 39, 41, 36, 36, 40, 41]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434771281Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"downstairs","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434791089Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"downstairs","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"[2]R flush_numbers[0..12]: [70, 76, 69, 72, 73, 78, 73, 79, 70, 70, 78, 79]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434809576Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"downstairs","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"[2]R generation[0..12]: [36, 40, 35, 38, 39, 40, 39, 41, 36, 36, 40, 41]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434827994Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"downstairs","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434847292Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"downstairs","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"Max found gen is 42","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434864909Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"downstairs","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"Generation requested: 42 >= found:42","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434881817Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"downstairs","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"Next flush: 80","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434898095Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"downstairs","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434917713Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.43494012Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434955698Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"client":"0","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434973836Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"client":"1","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.434990594Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"client":"2","":"downstairs","session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.435007922Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.435024009Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"8e8b8ea3-c28f-4e1d-86ed-cc93ddf3b28d is now active with session: 35a0a5af-cb3e-425c-8d7a-403220958dc0","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.435042217Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.435057365Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368,"session_id":"35a0a5af-cb3e-425c-8d7a-403220958dc0"} {"msg":"The guest has finished waiting for activation with:42","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.435073293Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368} {"msg":"Activated sub_volume 8e8b8ea3-c28f-4e1d-86ed-cc93ddf3b28d","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:31.435087272Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 116 len 20480 data: 8 7 6 6 6 02/30 Write block 32 len 40960 data: 2 2 3 4 6 6 7 7 7 6 03/30 Write block 34 len 12288 data: 4 5 7 04/30 Write block 421 len 32768 data: 5 5 5 7 7 6 8 7 05/30 Read block 515 len 12288 06/30 Read block 43 len 8192 07/30 Flush 08/30 Write block 232 len 4096 data: 5 09/30 Write block 451 len 20480 data: 6 6 6 6 7 10/30 Read block 307 len 20480 11/30 Write block 449 len 4096 data: 4 12/30 Write block 575 len 8192 data: 2 2 13/30 Write block 589 len 4096 data: 11 14/30 Write block 28 len 40960 data: 5 3 3 3 3 3 5 6 8 7 15/30 Read block 180 len 12288 16/30 Write block 515 len 24576 data: 3 3 6 5 5 5 17/30 Read block 154 len 8192 18/30 Read block 284 len 28672 19/30 Flush 20/30 Write block 33 len 24576 data: 4 6 7 9 8 8 21/30 Read block 209 len 16384 22/30 Write block 426 len 20480 data: 7 9 8 7 6 23/30 Flush 24/30 Write block 441 len 16384 data: 6 4 3 4 25/30 Read block 405 len 8192 26/30 Read block 185 len 4096 27/30 Read block 202 len 12288 28/30 Read block 184 len 8192 29/30 Read block 380 len 36864 30/30 Read block 284 len 4096 ---------------------------------------------------------------- Crucible gen:42 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Write 3 Done Done Done false 1003 Acked Write 8 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Read 2 Done Done Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 1 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Read 5 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Write 2 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Write 10 Done Sent Done false 1014 Acked Read 3 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 6 Done Sent Done false 1020 Acked Read 4 Done Sent Done false 1021 Acked Write 5 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Read 3 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Read 1 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 24 0 24 Done 30 6 30 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:31.453166054Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1368} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 70 70 70 F F F 1 020-039 42 40 42 82 76 82 F T F <--- 2 040-059 42 35 42 80 69 80 F T F <--- 3 060-079 38 38 38 72 72 72 F F F 4 080-099 39 39 39 73 73 73 F F F 5 100-119 42 40 42 80 78 80 F T F <--- 6 120-139 42 39 42 80 73 80 F T F <--- 7 140-159 41 41 41 79 79 79 F F F 8 160-179 36 36 36 70 70 70 F F F 9 180-199 36 36 36 70 70 70 F F F 10 200-219 40 40 40 78 78 78 F F F 11 220-239 42 41 42 81 79 81 F F F <--- 12 240-259 39 39 39 73 73 73 F F F 13 260-279 36 36 36 70 70 70 F F F 14 280-299 40 40 40 74 74 74 F F F 15 300-319 41 41 41 79 79 79 F F F 16 320-339 40 40 40 75 75 75 F F F 17 340-359 35 35 35 69 69 69 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 41 41 41 79 79 79 F F F 20 400-419 33 33 33 65 65 65 F F F 21 420-439 42 41 42 82 79 82 F T F <--- 22 440-459 42 40 42 81 76 81 T F T <--- 23 460-479 40 40 40 77 77 77 F F F 24 480-499 40 40 40 77 77 77 F F F 25 500-519 42 38 42 81 71 81 F F F <--- 26 520-539 42 38 42 81 71 81 F F F <--- 27 540-559 38 38 38 72 72 72 F F F 28 560-579 42 38 42 81 72 81 F F F <--- 29 580-599 42 31 42 81 63 81 F F F <--- Max gen: 42, Max flush: 82 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 21 Verifying data now No change after verify Loop: 21 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 36 36 36 70 70 70 F F F 1 020-039 42 42 42 82 82 82 F F F 2 040-059 42 42 42 80 80 80 F F F 3 060-079 38 38 38 72 72 72 F F F 4 080-099 39 39 39 73 73 73 F F F 5 100-119 42 42 42 80 80 80 F F F 6 120-139 42 42 42 80 80 80 F F F 7 140-159 41 41 41 79 79 79 F F F 8 160-179 36 36 36 70 70 70 F F F 9 180-199 36 36 36 70 70 70 F F F 10 200-219 40 40 40 78 78 78 F F F 11 220-239 42 42 42 81 81 81 F F F 12 240-259 39 39 39 73 73 73 F F F 13 260-279 36 36 36 70 70 70 F F F 14 280-299 40 40 40 74 74 74 F F F 15 300-319 41 41 41 79 79 79 F F F 16 320-339 40 40 40 75 75 75 F F F 17 340-359 35 35 35 69 69 69 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 41 41 41 79 79 79 F F F 20 400-419 33 33 33 65 65 65 F F F 21 420-439 42 42 42 82 82 82 F F F 22 440-459 43 43 43 83 83 83 F F F 23 460-479 40 40 40 77 77 77 F F F 24 480-499 40 40 40 77 77 77 F F F 25 500-519 42 42 42 81 81 81 F F F 26 520-539 42 42 42 81 81 81 F F F 27 540-559 38 38 38 72 72 72 F F F 28 560-579 42 42 42 81 81 81 F F F 29 580-599 42 42 42 81 81 81 F F F Max gen: 43, Max flush: 83 Begin loop 22 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.845443969Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.892929447Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.893565137Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.893613481Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.893629709Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.893651556Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376} {"msg":"Crucible 1ae71f69-6988-4c66-9f47-b78c8076a57b has session id: ee9c230f-7602-4042-9511-9a7d4c55f2c0","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.894155303Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"Upstairs opts: Upstairs UUID: 1ae71f69-6988-4c66-9f47-b78c8076a57b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.894218095Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"Crucible stats registered with UUID: 1ae71f69-6988-4c66-9f47-b78c8076a57b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.894242921Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:35.894259319Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.894468413Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"io task","client":"0","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.894516797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"io task","client":"2","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.894551293Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"io task","client":"1","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.894779604Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"task":"crutest"} {"msg":"The guest has requested activation with gen:44","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.89480876Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376} {"msg":"1ae71f69-6988-4c66-9f47-b78c8076a57b active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.894872092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.894893969Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"client":"0","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.894911787Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"client":"1","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.894929025Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"client":"2","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.894946263Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"io task","client":"2","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.894973879Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"io task","client":"0","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.895009885Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"io task","client":"1","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"ds_connection connected from Ok(127.0.0.1:39714)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.895054189Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"io task","client":"0","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"ds_connection connected from Ok(127.0.0.1:39454)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.895090145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"io task","client":"2","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"ds_connection connected from Ok(127.0.0.1:42796)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.89512537Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"io task","client":"1","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.896370943Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"client":"2","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.896417278Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"client":"0","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.896990795Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.89703158Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.897264581Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"client":"1","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"[0]R flush_numbers[0..12]: [70, 82, 80, 72, 73, 80, 80, 79, 70, 70, 78, 81]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.897793744Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"downstairs","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"[0]R generation[0..12]: [36, 42, 42, 38, 39, 42, 42, 41, 36, 36, 40, 42]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.89782646Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"downstairs","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.897848427Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"downstairs","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"[1]R flush_numbers[0..12]: [70, 82, 80, 72, 73, 80, 80, 79, 70, 70, 78, 81]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.897867705Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"downstairs","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"[1]R generation[0..12]: [36, 42, 42, 38, 39, 42, 42, 41, 36, 36, 40, 42]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.897886352Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"downstairs","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.897910319Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"downstairs","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"[2]R flush_numbers[0..12]: [70, 82, 80, 72, 73, 80, 80, 79, 70, 70, 78, 81]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.897929167Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"downstairs","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"[2]R generation[0..12]: [36, 42, 42, 38, 39, 42, 42, 41, 36, 36, 40, 42]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.897947615Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"downstairs","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.897967492Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"downstairs","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"Max found gen is 44","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.897991199Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"downstairs","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"Generation requested: 44 >= found:44","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-11-06T20:12:35.898021515Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"downstairs","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"Next flush: 84","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.898055641Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"downstairs","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.898081198Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"":"downstairs","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.898105715Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.898133741Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"client":"0","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.898155608Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"client":"1","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.898184575Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"client":"2","":"downstairs","session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.898211272Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.898242077Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"1ae71f69-6988-4c66-9f47-b78c8076a57b is now active with session: ee9c230f-7602-4042-9511-9a7d4c55f2c0","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.898266825Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.898285142Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376,"session_id":"ee9c230f-7602-4042-9511-9a7d4c55f2c0"} {"msg":"The guest has finished waiting for activation with:44","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.89830155Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376} {"msg":"Activated sub_volume 1ae71f69-6988-4c66-9f47-b78c8076a57b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:35.898315538Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 8 len 4096 data: 3 02/30 Read block 179 len 4096 03/30 Read block 87 len 4096 04/30 Read block 385 len 32768 05/30 Read block 132 len 12288 06/30 Read block 520 len 40960 07/30 Flush 08/30 Write block 279 len 32768 data: 5 4 4 3 4 4 4 4 09/30 Flush 10/30 Write block 353 len 4096 data: 3 11/30 Flush 12/30 Write block 114 len 12288 data: 6 6 9 13/30 Flush 14/30 Write block 590 len 12288 data: 10 7 6 15/30 Read block 411 len 32768 16/30 Read block 234 len 4096 17/30 Read block 420 len 40960 18/30 Read block 231 len 28672 19/30 Read block 502 len 32768 20/30 Read block 25 len 24576 21/30 Read block 539 len 36864 22/30 Write block 270 len 16384 data: 5 5 5 3 23/30 Write block 561 len 28672 data: 2 2 2 3 3 3 4 24/30 Read block 185 len 24576 25/30 Write block 478 len 32768 data: 5 6 4 5 4 4 3 3 26/30 Read block 242 len 20480 27/30 Write block 260 len 20480 data: 2 2 2 2 2 28/30 Read block 361 len 36864 29/30 Read block 512 len 36864 30/30 Read block 421 len 28672 ---------------------------------------------------------------- Crucible gen:44 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Read 3 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 8 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 3 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Read 1 Done Sent Done false 1016 Acked Read 10 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Read 8 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Read 6 Done Sent Done false 1024 Acked Write 8 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Write 5 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Read 7 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 26 0 26 Done 30 4 30 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) None Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:35.917814432Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1376} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 36 44 84 70 84 F T F <--- 1 020-039 42 42 42 82 82 82 F F F 2 040-059 42 42 42 80 80 80 F F F 3 060-079 38 38 38 72 72 72 F F F 4 080-099 39 39 39 73 73 73 F F F 5 100-119 44 42 44 87 80 87 F F F <--- 6 120-139 42 42 42 80 80 80 F F F 7 140-159 41 41 41 79 79 79 F F F 8 160-179 36 36 36 70 70 70 F F F 9 180-199 36 36 36 70 70 70 F F F 10 200-219 40 40 40 78 78 78 F F F 11 220-239 42 42 42 81 81 81 F F F 12 240-259 39 39 39 73 73 73 F F F 13 260-279 44 36 44 85 70 85 T F T <--- 14 280-299 44 40 44 85 74 85 F F F <--- 15 300-319 41 41 41 79 79 79 F F F 16 320-339 40 40 40 75 75 75 F F F 17 340-359 44 35 44 86 69 86 F F F <--- 18 360-379 28 28 28 57 57 57 F F F 19 380-399 41 41 41 79 79 79 F F F 20 400-419 33 33 33 65 65 65 F F F 21 420-439 42 42 42 82 82 82 F F F 22 440-459 43 43 43 83 83 83 F F F 23 460-479 40 40 40 77 77 77 T F T <--- 24 480-499 40 40 40 77 77 77 T F T <--- 25 500-519 42 42 42 81 81 81 F F F 26 520-539 42 42 42 81 81 81 F F F 27 540-559 38 38 38 72 72 72 F F F 28 560-579 42 42 42 81 81 81 T F T <--- 29 580-599 42 42 42 81 81 81 T F T <--- Max gen: 44, Max flush: 87 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 22 Verifying data now No change after verify Loop: 22 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 42 42 42 82 82 82 F F F 2 040-059 42 42 42 80 80 80 F F F 3 060-079 38 38 38 72 72 72 F F F 4 080-099 39 39 39 73 73 73 F F F 5 100-119 44 44 44 87 87 87 F F F 6 120-139 42 42 42 80 80 80 F F F 7 140-159 41 41 41 79 79 79 F F F 8 160-179 36 36 36 70 70 70 F F F 9 180-199 36 36 36 70 70 70 F F F 10 200-219 40 40 40 78 78 78 F F F 11 220-239 42 42 42 81 81 81 F F F 12 240-259 39 39 39 73 73 73 F F F 13 260-279 45 45 45 88 88 88 F F F 14 280-299 44 44 44 85 85 85 F F F 15 300-319 41 41 41 79 79 79 F F F 16 320-339 40 40 40 75 75 75 F F F 17 340-359 44 44 44 86 86 86 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 41 41 41 79 79 79 F F F 20 400-419 33 33 33 65 65 65 F F F 21 420-439 42 42 42 82 82 82 F F F 22 440-459 43 43 43 83 83 83 F F F 23 460-479 45 45 45 88 88 88 F F F 24 480-499 45 45 45 88 88 88 F F F 25 500-519 42 42 42 81 81 81 F F F 26 520-539 42 42 42 81 81 81 F F F 27 540-559 38 38 38 72 72 72 F F F 28 560-579 45 45 45 88 88 88 F F F 29 580-599 45 45 45 88 88 88 F F F Max gen: 45, Max flush: 88 Begin loop 23 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.304407484Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.350345276Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.350999284Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.351052877Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.351069675Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.351092022Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384} {"msg":"Crucible f313ec5c-fc02-46ea-adf1-05c183e44277 has session id: a6b9bdbf-8d85-4081-a022-e8e518152af8","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.351527367Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"Upstairs opts: Upstairs UUID: f313ec5c-fc02-46ea-adf1-05c183e44277, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.351577341Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"Crucible stats registered with UUID: f313ec5c-fc02-46ea-adf1-05c183e44277","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.351608197Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:40.351636523Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.351850596Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"io task","client":"0","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.351896181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"io task","client":"2","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.351926317Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"io task","client":"1","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.352094736Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"task":"crutest"} {"msg":"The guest has requested activation with gen:46","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.352130561Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384} {"msg":"f313ec5c-fc02-46ea-adf1-05c183e44277 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.352196783Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.352237208Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"client":"0","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.352271143Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"client":"1","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.352305269Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"client":"2","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.352338475Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"io task","client":"2","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.352369531Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"io task","client":"0","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.352402097Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"io task","client":"1","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"ds_connection connected from Ok(127.0.0.1:52393)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.352434273Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"io task","client":"2","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"ds_connection connected from Ok(127.0.0.1:59886)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.352466969Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"io task","client":"0","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"ds_connection connected from Ok(127.0.0.1:48243)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.352567826Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"io task","client":"1","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.353676066Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"client":"2","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.353787543Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"client":"1","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.353814689Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"client":"0","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354289019Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354391327Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"[0]R flush_numbers[0..12]: [84, 82, 80, 72, 73, 87, 80, 79, 70, 70, 78, 81]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354430071Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"downstairs","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"[0]R generation[0..12]: [44, 42, 42, 38, 39, 44, 42, 41, 36, 36, 40, 42]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354451409Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"downstairs","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354471816Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"downstairs","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"[1]R flush_numbers[0..12]: [84, 82, 80, 72, 73, 87, 80, 79, 70, 70, 78, 81]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354490634Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"downstairs","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"[1]R generation[0..12]: [44, 42, 42, 38, 39, 44, 42, 41, 36, 36, 40, 42]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354508892Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"downstairs","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354528529Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"downstairs","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"[2]R flush_numbers[0..12]: [84, 82, 80, 72, 73, 87, 80, 79, 70, 70, 78, 81]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354546997Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"downstairs","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"[2]R generation[0..12]: [44, 42, 42, 38, 39, 44, 42, 41, 36, 36, 40, 42]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354565354Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"downstairs","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354585212Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"downstairs","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"Max found gen is 46","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.35460154Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"downstairs","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"Generation requested: 46 >= found:46","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354621227Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"downstairs","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"Next flush: 89","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-11-06T20:12:40.354637476Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"downstairs","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354660522Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"":"downstairs","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.35468141Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354697028Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"client":"0","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354713986Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"client":"1","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354730914Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"client":"2","":"downstairs","session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354747651Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354763809Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"f313ec5c-fc02-46ea-adf1-05c183e44277 is now active with session: a6b9bdbf-8d85-4081-a022-e8e518152af8","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354781697Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354796875Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384,"session_id":"a6b9bdbf-8d85-4081-a022-e8e518152af8"} {"msg":"The guest has finished waiting for activation with:46","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354812843Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384} {"msg":"Activated sub_volume f313ec5c-fc02-46ea-adf1-05c183e44277","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:40.354826752Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 505 len 40960 data: 3 3 3 4 4 4 4 4 4 3 02/30 Write block 332 len 28672 data: 5 5 5 4 4 4 5 03/30 Read block 540 len 12288 04/30 Write block 112 len 32768 data: 5 6 7 7 10 8 7 7 05/30 Write block 454 len 24576 data: 7 8 7 6 4 4 06/30 Write block 128 len 8192 data: 8 10 07/30 Flush 08/30 Write block 448 len 36864 data: 4 5 5 7 7 7 8 9 8 09/30 Write block 111 len 40960 data: 5 6 7 8 8 11 9 8 8 7 10/30 Write block 175 len 4096 data: 3 11/30 Write block 484 len 28672 data: 4 4 3 3 4 3 2 12/30 Read block 505 len 28672 13/30 Read block 532 len 16384 14/30 Flush 15/30 Write block 442 len 40960 data: 5 4 5 4 3 3 5 6 6 8 16/30 Read block 550 len 32768 17/30 Read block 247 len 12288 18/30 Read block 288 len 16384 19/30 Read block 176 len 40960 20/30 Write block 266 len 36864 data: 4 5 4 5 6 6 6 4 2 21/30 Read block 107 len 40960 22/30 Read block 579 len 32768 23/30 Flush 24/30 Write block 179 len 40960 data: 8 8 8 8 8 4 4 5 4 5 25/30 Read block 180 len 4096 26/30 Write block 116 len 4096 data: 12 27/30 Read block 467 len 16384 28/30 Read block 94 len 20480 29/30 Write block 220 len 8192 data: 2 2 30/30 Write block 116 len 20480 data: 13 10 9 9 8 ---------------------------------------------------------------- Crucible gen:46 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Write 8 Done Done Done false 1004 Acked Write 6 Done Sent Done false 1005 Acked Write 2 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 9 Done Sent Done false 1008 Acked Write 10 Done Sent Done false 1009 Acked Write 1 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Read 3 Done Sent Done false 1017 Acked Read 4 Done Sent Done false 1018 Acked Read 10 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Read 10 Done Sent Done false 1021 Acked Read 8 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 10 Done Sent Done false 1024 Acked Read 1 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Write 2 Done Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 26 2 29 Done 29 4 28 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:40.375136143Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1384} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 42 42 42 82 82 82 F F F 2 040-059 42 42 42 80 80 80 F F F 3 060-079 38 38 38 72 72 72 F F F 4 080-099 39 39 39 73 73 73 F F F 5 100-119 46 44 46 90 87 90 T T T <--- 6 120-139 46 42 46 90 80 90 T F T <--- 7 140-159 41 41 41 79 79 79 F F F 8 160-179 46 36 46 90 70 90 T F T <--- 9 180-199 36 36 36 70 70 70 T F T <--- 10 200-219 40 40 40 78 78 78 F F F 11 220-239 42 42 42 81 81 81 T F T <--- 12 240-259 39 39 39 73 73 73 F F F 13 260-279 46 45 46 91 88 91 F F F <--- 14 280-299 44 44 44 85 85 85 F F F 15 300-319 41 41 41 79 79 79 F F F 16 320-339 46 40 46 89 75 89 F T F <--- 17 340-359 44 44 44 86 86 86 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 41 41 41 79 79 79 F F F 20 400-419 33 33 33 65 65 65 F F F 21 420-439 42 42 42 82 82 82 F F F 22 440-459 46 43 46 91 83 91 F F F <--- 23 460-479 45 45 45 88 88 88 F F F 24 480-499 46 45 46 90 88 90 F F F <--- 25 500-519 46 42 46 89 81 89 F T F <--- 26 520-539 42 42 42 81 81 81 F F F 27 540-559 38 38 38 72 72 72 F F F 28 560-579 45 45 45 88 88 88 F F F 29 580-599 45 45 45 88 88 88 F F F Max gen: 46, Max flush: 91 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 23 Verifying data now No change after verify Loop: 23 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 42 42 42 82 82 82 F F F 2 040-059 42 42 42 80 80 80 F F F 3 060-079 38 38 38 72 72 72 F F F 4 080-099 39 39 39 73 73 73 F F F 5 100-119 47 47 47 92 92 92 F F F 6 120-139 47 47 47 92 92 92 F F F 7 140-159 41 41 41 79 79 79 F F F 8 160-179 47 47 47 92 92 92 F F F 9 180-199 47 47 47 92 92 92 F F F 10 200-219 40 40 40 78 78 78 F F F 11 220-239 47 47 47 92 92 92 F F F 12 240-259 39 39 39 73 73 73 F F F 13 260-279 46 46 46 91 91 91 F F F 14 280-299 44 44 44 85 85 85 F F F 15 300-319 41 41 41 79 79 79 F F F 16 320-339 46 46 46 89 89 89 F F F 17 340-359 44 44 44 86 86 86 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 41 41 41 79 79 79 F F F 20 400-419 33 33 33 65 65 65 F F F 21 420-439 42 42 42 82 82 82 F F F 22 440-459 46 46 46 91 91 91 F F F 23 460-479 45 45 45 88 88 88 F F F 24 480-499 46 46 46 90 90 90 F F F 25 500-519 46 46 46 89 89 89 F F F 26 520-539 42 42 42 81 81 81 F F F 27 540-559 38 38 38 72 72 72 F F F 28 560-579 45 45 45 88 88 88 F F F 29 580-599 45 45 45 88 88 88 F F F Max gen: 47, Max flush: 92 Begin loop 24 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.794746814Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.840071153Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.840699324Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.840748378Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.840764936Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.840786973Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392} {"msg":"Crucible 5ee915a4-7ce4-4d3b-b463-d3f98ac7aa3b has session id: 61999e50-a162-4a56-bc4d-19d13e604861","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.841390137Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"Upstairs opts: Upstairs UUID: 5ee915a4-7ce4-4d3b-b463-d3f98ac7aa3b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.84144764Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"Crucible stats registered with UUID: 5ee915a4-7ce4-4d3b-b463-d3f98ac7aa3b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.841475767Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:44.841494004Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.84168463Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"io task","client":"2","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.841725565Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"io task","client":"1","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.841744922Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"io task","client":"0","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.841977953Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"task":"crutest"} {"msg":"The guest has requested activation with gen:48","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.842018498Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392} {"msg":"5ee915a4-7ce4-4d3b-b463-d3f98ac7aa3b active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.84208135Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.842101448Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"client":"0","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.842124335Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"client":"1","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.842204415Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"client":"2","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.842245399Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"io task","client":"2","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.842277646Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"io task","client":"0","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.842308722Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"io task","client":"1","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"ds_connection connected from Ok(127.0.0.1:57781)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.8423988Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"io task","client":"2","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"ds_connection connected from Ok(127.0.0.1:56616)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.842442635Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"io task","client":"0","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"ds_connection connected from Ok(127.0.0.1:34021)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.842472471Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"io task","client":"1","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.843616147Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"client":"1","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.843662531Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"client":"0","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.844216901Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.844313919Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.844339576Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"client":"2","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"[0]R flush_numbers[0..12]: [84, 82, 80, 72, 73, 92, 92, 79, 92, 92, 78, 92]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.844878268Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"downstairs","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"[0]R generation[0..12]: [44, 42, 42, 38, 39, 47, 47, 41, 47, 47, 40, 47]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.844901845Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"downstairs","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.844921972Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"downstairs","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"[1]R flush_numbers[0..12]: [84, 82, 80, 72, 73, 92, 92, 79, 92, 92, 78, 92]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.84494096Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"downstairs","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"[1]R generation[0..12]: [44, 42, 42, 38, 39, 47, 47, 41, 47, 47, 40, 47]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.844959637Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"downstairs","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.844979735Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"downstairs","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"[2]R flush_numbers[0..12]: [84, 82, 80, 72, 73, 92, 92, 79, 92, 92, 78, 92]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845002832Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"downstairs","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"[2]R generation[0..12]: [44, 42, 42, 38, 39, 47, 47, 41, 47, 47, 40, 47]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.84502147Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"downstairs","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845041227Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"downstairs","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"Max found gen is 48","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845057535Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"downstairs","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"Generation requested: 48 >= found:48","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-11-06T20:12:44.845075293Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"downstairs","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"Next flush: 93","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845095211Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"downstairs","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845111598Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"":"downstairs","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845128636Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845143844Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"client":"0","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845160642Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"client":"1","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.84517736Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"client":"2","":"downstairs","session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845194398Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845210286Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"5ee915a4-7ce4-4d3b-b463-d3f98ac7aa3b is now active with session: 61999e50-a162-4a56-bc4d-19d13e604861","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845228214Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845247471Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392,"session_id":"61999e50-a162-4a56-bc4d-19d13e604861"} {"msg":"The guest has finished waiting for activation with:48","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845265989Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392} {"msg":"Activated sub_volume 5ee915a4-7ce4-4d3b-b463-d3f98ac7aa3b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:44.845280277Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 237 len 32768 data: 4 5 5 4 5 5 6 6 02/30 Flush 03/30 Write block 311 len 28672 data: 7 6 6 6 4 4 4 04/30 Write block 399 len 8192 data: 4 4 05/30 Write block 419 len 16384 data: 5 5 6 6 06/30 Read block 60 len 4096 07/30 Write block 114 len 24576 data: 9 9 14 11 10 10 08/30 Flush 09/30 Write block 304 len 20480 data: 5 6 5 7 6 10/30 Write block 577 len 4096 data: 2 11/30 Read block 89 len 24576 12/30 Write block 104 len 32768 data: 3 2 2 3 3 4 4 6 13/30 Read block 344 len 12288 14/30 Read block 94 len 32768 15/30 Write block 454 len 28672 data: 9 10 9 7 5 5 2 16/30 Read block 271 len 20480 17/30 Read block 233 len 8192 18/30 Read block 369 len 32768 19/30 Write block 541 len 8192 data: 5 7 20/30 Read block 153 len 36864 21/30 Write block 60 len 12288 data: 3 3 3 22/30 Read block 427 len 32768 23/30 Read block 481 len 16384 24/30 Read block 5 len 4096 25/30 Write block 550 len 16384 data: 6 5 5 6 26/30 Write block 282 len 8192 data: 4 5 27/30 Read block 419 len 4096 28/30 Read block 14 len 36864 29/30 Write block 110 len 8192 data: 5 7 30/30 Write block 181 len 12288 data: 9 9 9 ---------------------------------------------------------------- Crucible gen:48 GIO:true work queues: Upstairs:0 downstairs:22 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1008 Acked Write 5 Done Done Done false 1009 Acked Write 1 Done Done Done false 1010 Acked Read 6 Done Done Sent false 1011 Acked Write 8 Done Done Sent false 1012 Acked Read 3 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Read 5 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Write 2 Done Done Sent false 1019 Acked Read 9 Done Done Sent false 1020 Acked Write 3 Done Done Sent false 1021 Acked Read 8 Done Done Sent false 1022 Acked Read 4 Done Done Sent false 1023 Acked Read 1 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Write 2 Done Done Sent false 1026 Acked Read 1 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Write 2 Done Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 2 20 23 Done 21 20 2 43 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1007)) Some(JobId(1007)) Some(JobId(1007)) Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:44.86351087Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1392} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 42 42 42 82 82 82 F F F 2 040-059 42 42 42 80 80 80 F F F 3 060-079 38 38 38 72 72 72 T T F <--- 4 080-099 39 39 39 73 73 73 F F F 5 100-119 48 48 48 94 94 94 T T F <--- 6 120-139 47 47 47 92 92 92 F F F 7 140-159 41 41 41 79 79 79 F F F 8 160-179 47 47 47 92 92 92 F F F 9 180-199 47 47 47 92 92 92 T T F <--- 10 200-219 40 40 40 78 78 78 F F F 11 220-239 48 48 48 93 93 93 F F F 12 240-259 48 48 48 93 93 93 F F F 13 260-279 46 46 46 91 91 91 F F F 14 280-299 44 44 44 85 85 85 T T F <--- 15 300-319 48 48 48 94 94 94 T T T 16 320-339 46 46 46 89 89 89 F F F 17 340-359 44 44 44 86 86 86 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 48 48 48 94 94 94 F F F 20 400-419 48 48 48 94 94 94 F F F 21 420-439 48 48 48 94 94 94 F F F 22 440-459 46 46 46 91 91 91 T T F <--- 23 460-479 45 45 45 88 88 88 T T F <--- 24 480-499 46 46 46 90 90 90 F F F 25 500-519 46 46 46 89 89 89 F F F 26 520-539 42 42 42 81 81 81 F F F 27 540-559 38 38 38 72 72 72 T T F <--- 28 560-579 45 45 45 88 88 88 T T T 29 580-599 45 45 45 88 88 88 F F F Max gen: 48, Max flush: 94 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 24 Verifying data now No change after verify Loop: 24 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 42 42 42 82 82 82 F F F 2 040-059 42 42 42 80 80 80 F F F 3 060-079 49 49 49 95 95 95 F F F 4 080-099 39 39 39 73 73 73 F F F 5 100-119 49 49 49 95 95 95 F F F 6 120-139 47 47 47 92 92 92 F F F 7 140-159 41 41 41 79 79 79 F F F 8 160-179 47 47 47 92 92 92 F F F 9 180-199 49 49 49 95 95 95 F F F 10 200-219 40 40 40 78 78 78 F F F 11 220-239 48 48 48 93 93 93 F F F 12 240-259 48 48 48 93 93 93 F F F 13 260-279 46 46 46 91 91 91 F F F 14 280-299 49 49 49 95 95 95 F F F 15 300-319 49 49 49 95 95 95 F F F 16 320-339 46 46 46 89 89 89 F F F 17 340-359 44 44 44 86 86 86 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 48 48 48 94 94 94 F F F 20 400-419 48 48 48 94 94 94 F F F 21 420-439 48 48 48 94 94 94 F F F 22 440-459 49 49 49 95 95 95 F F F 23 460-479 49 49 49 95 95 95 F F F 24 480-499 46 46 46 90 90 90 F F F 25 500-519 46 46 46 89 89 89 F F F 26 520-539 42 42 42 81 81 81 F F F 27 540-559 49 49 49 95 95 95 F F F 28 560-579 49 49 49 95 95 95 F F F 29 580-599 45 45 45 88 88 88 F F F Max gen: 49, Max flush: 95 Begin loop 25 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.256441414Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.303374521Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.304031768Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.304080392Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.30409691Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.304124566Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400} {"msg":"Crucible a43a57ef-4a72-45d0-840d-75d15e3efcce has session id: 7aa19a60-ad91-46ac-9692-e3c435285e5b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.3046491Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"Upstairs opts: Upstairs UUID: a43a57ef-4a72-45d0-840d-75d15e3efcce, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.304707943Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"Crucible stats registered with UUID: a43a57ef-4a72-45d0-840d-75d15e3efcce","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.304737689Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:59.304762186Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.30497015Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"2","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.305016164Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"0","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.305037201Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.30520725Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"task":"crutest"} {"msg":"The guest has requested activation with gen:50","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.305234776Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400} {"msg":"a43a57ef-4a72-45d0-840d-75d15e3efcce active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.305320385Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.305345473Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"client":"0","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.30536329Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"client":"1","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.305380208Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"client":"2","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.305396936Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"2","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.305414184Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"0","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.305436111Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"ds_connection connected from Ok(127.0.0.1:42214)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.305564425Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"2","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"ds_connection connected from Ok(127.0.0.1:52480)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.305609399Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"0","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:59.305653314Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:59.305698858Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:12:59.305733473Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.305864117Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.305902342Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.306764824Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"client":"0","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.306858552Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"client":"2","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.307373107Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:12:59.307461466Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.307331882Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"ds_connection connected from Ok(127.0.0.1:47686)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.307475853Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"io task","client":"1","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.308870917Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"client":"1","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"[0]R flush_numbers[0..12]: [84, 82, 80, 95, 73, 95, 92, 79, 92, 95, 78, 93]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309501308Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"downstairs","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"[0]R generation[0..12]: [44, 42, 42, 49, 39, 49, 47, 41, 47, 49, 40, 48]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309537933Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"downstairs","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.30956468Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"downstairs","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"[1]R flush_numbers[0..12]: [84, 82, 80, 95, 73, 95, 92, 79, 92, 95, 78, 93]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309588347Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"downstairs","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"[1]R generation[0..12]: [44, 42, 42, 49, 39, 49, 47, 41, 47, 49, 40, 48]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309611584Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"downstairs","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309636191Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"downstairs","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"[2]R flush_numbers[0..12]: [84, 82, 80, 95, 73, 95, 92, 79, 92, 95, 78, 93]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309659218Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"downstairs","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"[2]R generation[0..12]: [44, 42, 42, 49, 39, 49, 47, 41, 47, 49, 40, 48]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309682265Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"downstairs","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309706432Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"downstairs","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"Max found gen is 50","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309732939Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"downstairs",Wait for a query_work_queue command to finish before sending IO "":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"Generation requested: 50 >= found:50","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309761835Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"downstairs","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"Next flush: 96","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309782153Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"downstairs","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.3098021Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"":"downstairs","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309824348Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309843755Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"client":"0","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309864972Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"client":"1","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.30988646Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"client":"2","":"downstairs","session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309908077Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309928054Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"a43a57ef-4a72-45d0-840d-75d15e3efcce is now active with session: 7aa19a60-ad91-46ac-9692-e3c435285e5b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309954521Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.309973719Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400,"session_id":"7aa19a60-ad91-46ac-9692-e3c435285e5b"} {"msg":"The guest has finished waiting for activation with:50","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.310010174Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400} {"msg":"Activated sub_volume a43a57ef-4a72-45d0-840d-75d15e3efcce","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:09.310031191Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 88 len 36864 data: 4 5 4 3 3 3 3 4 4 03/30 Flush 04/30 Write block 521 len 8192 data: 6 4 05/30 Read block 258 len 16384 06/30 Write block 131 len 8192 data: 8 8 07/30 Write block 120 len 8192 data: 9 5 08/30 Read block 487 len 32768 09/30 Read block 45 len 28672 10/30 Read block 207 len 24576 11/30 Read block 129 len 20480 12/30 Read block 332 len 16384 13/30 Read block 555 len 4096 14/30 Write block 95 len 32768 data: 5 5 5 3 4 3 2 3 15/30 Read block 414 len 40960 16/30 Read block 169 len 24576 17/30 Write block 577 len 4096 data: 3 18/30 Read block 140 len 24576 19/30 Write block 271 len 36864 data: 7 7 5 3 2 4 6 6 6 20/30 Write block 168 len 8192 data: 4 4 21/30 Flush 22/30 Write block 595 len 16384 data: 4 6 4 3 23/30 Write block 86 len 36864 data: 4 4 5 6 5 4 4 4 4 24/30 Read block 1 len 32768 25/30 Read block 430 len 20480 26/30 Read block 369 len 20480 27/30 Write block 89 len 16384 data: 7 6 5 5 28/30 Write block 561 len 12288 data: 3 3 3 29/30 Write block 593 len 12288 data: 4 4 5 30/30 Read block 163 len 8192 ---------------------------------------------------------------- Crucible gen:50 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 2 Done Sent Done false 1004 Acked Read 4 Done Sent Done false 1005 Acked Write 2 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Read 7 Done Sent Done false 1009 Acked Read 6 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Read 4 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Write 2 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 4 Done Sent Done false 1022 Acked Write 9 Done Sent Done false 1023 Acked Read 8 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Write 3 Sent Sent Done false 1029 Acked Read 2 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 27 0 29 Done 25 0 27 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1002)) Some(JobId(1020)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:09.328742044Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1400} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 42 42 42 82 82 82 F F F 2 040-059 42 42 42 80 80 80 F F F 3 060-079 49 49 49 95 95 95 F F F 4 080-099 50 50 50 98 97 98 T F T <--- 5 100-119 50 49 50 98 95 98 F F F <--- 6 120-139 50 47 50 98 92 98 F F F <--- 7 140-159 41 41 41 79 79 79 F F F 8 160-179 50 47 50 98 92 98 F F F <--- 9 180-199 49 49 49 95 95 95 F F F 10 200-219 40 40 40 78 78 78 F F F 11 220-239 48 48 48 93 93 93 F F F 12 240-259 48 48 48 93 93 93 F F F 13 260-279 50 46 50 98 91 98 F F F <--- 14 280-299 49 49 49 95 95 95 F F F 15 300-319 49 49 49 95 95 95 F F F 16 320-339 46 46 46 89 89 89 F F F 17 340-359 44 44 44 86 86 86 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 48 48 48 94 94 94 F F F 20 400-419 48 48 48 94 94 94 F F F 21 420-439 48 48 48 94 94 94 F F F 22 440-459 49 49 49 95 95 95 F F F 23 460-479 49 49 49 95 95 95 F F F 24 480-499 46 46 46 90 90 90 F F F 25 500-519 46 46 46 89 89 89 F F F 26 520-539 50 42 50 98 81 98 F F F <--- 27 540-559 49 49 49 95 95 95 F F F 28 560-579 50 49 50 98 95 98 T F T <--- 29 580-599 45 45 45 88 88 88 T F T <--- Max gen: 50, Max flush: 98 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 25 Verifying data now No change after verify Loop: 25 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 42 42 42 82 82 82 F F F 2 040-059 42 42 42 80 80 80 F F F 3 060-079 49 49 49 95 95 95 F F F 4 080-099 51 51 51 99 99 99 F F F 5 100-119 50 50 50 98 98 98 F F F 6 120-139 50 50 50 98 98 98 F F F 7 140-159 41 41 41 79 79 79 F F F 8 160-179 50 50 50 98 98 98 F F F 9 180-199 49 49 49 95 95 95 F F F 10 200-219 40 40 40 78 78 78 F F F 11 220-239 48 48 48 93 93 93 F F F 12 240-259 48 48 48 93 93 93 F F F 13 260-279 50 50 50 98 98 98 F F F 14 280-299 49 49 49 95 95 95 F F F 15 300-319 49 49 49 95 95 95 F F F 16 320-339 46 46 46 89 89 89 F F F 17 340-359 44 44 44 86 86 86 F F F 18 360-379 28 28 28 57 57 57 F F F 19 380-399 48 48 48 94 94 94 F F F 20 400-419 48 48 48 94 94 94 F F F 21 420-439 48 48 48 94 94 94 F F F 22 440-459 49 49 49 95 95 95 F F F 23 460-479 49 49 49 95 95 95 F F F 24 480-499 46 46 46 90 90 90 F F F 25 500-519 46 46 46 89 89 89 F F F 26 520-539 50 50 50 98 98 98 F F F 27 540-559 49 49 49 95 95 95 F F F 28 560-579 51 51 51 99 99 99 F F F 29 580-599 51 51 51 99 99 99 F F F Max gen: 51, Max flush: 99 Begin loop 26 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.714386326Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.76054833Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.761196148Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.761244712Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.761266209Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.761289396Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408} {"msg":"Crucible f9905b22-9eb1-461c-942b-0fc0efbd82f6 has session id: 6a284c4d-784d-49cc-ab71-712cc326c6c1","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.761819379Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"Upstairs opts: Upstairs UUID: f9905b22-9eb1-461c-942b-0fc0efbd82f6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.761875432Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"Crucible stats registered with UUID: f9905b22-9eb1-461c-942b-0fc0efbd82f6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.761907418Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:13.761933345Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762171825Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"io task","client":"0","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762220559Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"io task","client":"2","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762254045Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"io task","client":"1","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762443831Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"task":"crutest"} {"msg":"The guest has requested activation with gen:52","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762472787Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408} {"msg":"f9905b22-9eb1-461c-942b-0fc0efbd82f6 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762538849Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762559306Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"client":"0","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762576974Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"client":"1","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762602361Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"client":"2","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762630597Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"io task","client":"2","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762658994Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"io task","client":"0","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762702218Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"io task","client":"1","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"ds_connection connected from Ok(127.0.0.1:63623)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762737934Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"io task","client":"2","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"ds_connection connected from Ok(127.0.0.1:64256)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.76276989Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"io task","client":"0","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"ds_connection connected from Ok(127.0.0.1:52641)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.762803806Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"io task","client":"1","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.76403487Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"client":"0","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.764075815Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"client":"1","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.764096433Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"client":"2","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.764573123Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.764686708Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"[0]R flush_numbers[0..12]: [84, 82, 80, 95, 99, 98, 98, 79, 98, 95, 78, 93]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.76475538Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"downstairs","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"[0]R generation[0..12]: [44, 42, 42, 49, 51, 50, 50, 41, 50, 49, 40, 48]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.764778727Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"downstairs","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.764799104Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"downstairs","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"[1]R flush_numbers[0..12]: [84, 82, 80, 95, 99, 98, 98, 79, 98, 95, 78, 93]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.764821891Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"downstairs","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"[1]R generation[0..12]: [44, 42, 42, 49, 51, 50, 50, 41, 50, 49, 40, 48]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.764840929Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"downstairs","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.764860756Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"downstairs","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"[2]R flush_numbers[0..12]: [84, 82, 80, 95, 99, 98, 98, 79, 98, 95, 78, 93]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.764879444Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"downstairs","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"[2]R generation[0..12]: [44, 42, 42, 49, 51, 50, 50, 41, 50, 49, 40, 48]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.764897922Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"downstairs","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.764927798Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"downstairs","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"Max found gen is 52","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.764957244Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"downstairs","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"Generation requested: 52 >= found:52","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-11-06T20:13:13.764975632Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"downstairs","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"Next flush: 100","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.765004448Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"downstairs","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.765027155Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"":"downstairs","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.765044803Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.765060181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"client":"0","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.765076919Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"client":"1","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.765093307Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"client":"2","":"downstairs","session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.765112755Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.765128982Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"f9905b22-9eb1-461c-942b-0fc0efbd82f6 is now active with session: 6a284c4d-784d-49cc-ab71-712cc326c6c1","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.76514688Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.765162088Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408,"session_id":"6a284c4d-784d-49cc-ab71-712cc326c6c1"} {"msg":"The guest has finished waiting for activation with:52","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.765178206Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408} {"msg":"Activated sub_volume f9905b22-9eb1-461c-942b-0fc0efbd82f6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:13.765192185Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 436 len 40960 data: 5 4 4 6 6 7 6 5 6 5 03/30 Flush 04/30 Write block 122 len 8192 data: 4 5 05/30 Write block 288 len 28672 data: 3 3 4 3 4 6 5 06/30 Read block 356 len 36864 07/30 Read block 416 len 24576 08/30 Read block 412 len 16384 09/30 Write block 50 len 28672 data: 3 3 2 3 3 2 2 10/30 Read block 292 len 20480 11/30 Read block 552 len 16384 12/30 Read block 391 len 36864 13/30 Read block 434 len 24576 14/30 Read block 426 len 40960 15/30 Flush 16/30 Write block 439 len 28672 data: 7 7 8 7 6 7 6 17/30 Write block 547 len 28672 data: 7 8 6 7 6 6 7 18/30 Flush 19/30 Write block 401 len 24576 data: 6 5 4 4 4 3 20/30 Flush 21/30 Write block 235 len 32768 data: 6 5 5 6 6 5 6 6 22/30 Write block 510 len 40960 data: 5 5 5 5 4 4 4 7 6 6 23/30 Read block 318 len 32768 24/30 Flush 25/30 Write block 202 len 28672 data: 5 6 7 7 6 6 4 26/30 Write block 370 len 4096 data: 2 27/30 Write block 289 len 12288 data: 4 5 4 28/30 Write block 355 len 12288 data: 4 4 3 29/30 Read block 263 len 28672 30/30 Read block 98 len 24576 ---------------------------------------------------------------- Crucible gen:52 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 10 Done Done Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 2 Done Sent Done false 1004 Acked Write 7 Done Sent Done false 1005 Acked Read 9 Done Sent Done false 1006 Acked Read 6 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Write 7 Done Sent Done false 1009 Acked Read 5 Done Sent Done false 1010 Acked Read 4 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Read 10 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 7 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 8 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 7 Done Sent Done false 1025 Acked Write 1 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 0 28 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1000)) Some(JobId(1023)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:13.785104965Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1408} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 42 42 42 82 82 82 F F F 2 040-059 52 42 52 102 80 102 F F F <--- 3 060-079 49 49 49 95 95 95 F F F 4 080-099 51 51 51 99 99 99 F F F 5 100-119 50 50 50 98 98 98 F F F 6 120-139 52 50 52 102 98 102 F F F <--- 7 140-159 41 41 41 79 79 79 F F F 8 160-179 50 50 50 98 98 98 F F F 9 180-199 49 49 49 95 95 95 F F F 10 200-219 40 40 40 78 78 78 T F T <--- 11 220-239 52 48 52 105 93 105 F F F <--- 12 240-259 52 48 52 105 93 105 F F F <--- 13 260-279 50 50 50 98 98 98 F F F 14 280-299 52 49 52 102 95 102 T F T <--- 15 300-319 49 49 49 95 95 95 F F F 16 320-339 46 46 46 89 89 89 F F F 17 340-359 44 44 44 86 86 86 T F T <--- 18 360-379 28 28 28 57 57 57 T F T <--- 19 380-399 48 48 48 94 94 94 F F F 20 400-419 52 48 52 104 94 104 F F F <--- 21 420-439 52 48 52 103 94 103 F T F <--- 22 440-459 52 49 52 103 95 103 F T F <--- 23 460-479 49 49 49 95 95 95 F F F 24 480-499 46 46 46 90 90 90 F F F 25 500-519 52 46 52 105 89 105 F F F <--- 26 520-539 50 50 50 98 98 98 F F F 27 540-559 52 49 52 103 95 103 F F F <--- 28 560-579 51 51 51 99 99 99 F F F 29 580-599 51 51 51 99 99 99 F F F Max gen: 52, Max flush: 105 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 26 Verifying data now No change after verify Loop: 26 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 42 42 42 82 82 82 F F F 2 040-059 52 52 52 102 102 102 F F F 3 060-079 49 49 49 95 95 95 F F F 4 080-099 51 51 51 99 99 99 F F F 5 100-119 50 50 50 98 98 98 F F F 6 120-139 52 52 52 102 102 102 F F F 7 140-159 41 41 41 79 79 79 F F F 8 160-179 50 50 50 98 98 98 F F F 9 180-199 49 49 49 95 95 95 F F F 10 200-219 53 53 53 106 106 106 F F F 11 220-239 52 52 52 105 105 105 F F F 12 240-259 52 52 52 105 105 105 F F F 13 260-279 50 50 50 98 98 98 F F F 14 280-299 53 53 53 106 106 106 F F F 15 300-319 49 49 49 95 95 95 F F F 16 320-339 46 46 46 89 89 89 F F F 17 340-359 53 53 53 106 106 106 F F F 18 360-379 53 53 53 106 106 106 F F F 19 380-399 48 48 48 94 94 94 F F F 20 400-419 52 52 52 104 104 104 F F F 21 420-439 52 52 52 103 103 103 F F F 22 440-459 52 52 52 103 103 103 F F F 23 460-479 49 49 49 95 95 95 F F F 24 480-499 46 46 46 90 90 90 F F F 25 500-519 52 52 52 105 105 105 F F F 26 520-539 50 50 50 98 98 98 F F F 27 540-559 52 52 52 103 103 103 F F F 28 560-579 51 51 51 99 99 99 F F F 29 580-599 51 51 51 99 99 99 F F F Max gen: 53, Max flush: 106 Begin loop 27 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.186541107Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.232111936Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.232718539Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.232762124Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.232777872Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.232799699Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416} {"msg":"Crucible 931cb81c-6742-4b0f-a14f-1d4792ef2a02 has session id: 3ddf5b7c-216a-4969-97eb-d84883cdff3b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.233364108Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"Upstairs opts: Upstairs UUID: 931cb81c-6742-4b0f-a14f-1d4792ef2a02, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.233420401Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"Crucible stats registered with UUID: 931cb81c-6742-4b0f-a14f-1d4792ef2a02","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.233450077Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:18.233475904Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.233642353Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"io task","client":"0","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.233690257Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"io task","client":"2","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.233721553Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"io task","client":"1","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.233864145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"task":"crutest"} {"msg":"The guest has requested activation with gen:54","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.23390709Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416} {"msg":"931cb81c-6742-4b0f-a14f-1d4792ef2a02 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.233943225Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.233960823Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"client":"0","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.233978701Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"client":"1","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.234004627Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"client":"2","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.234055701Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"io task","client":"2","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.234107664Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"io task","client":"0","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.23414389Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"io task","client":"1","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"ds_connection connected from Ok(127.0.0.1:58069)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.234215171Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"io task","client":"0","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"ds_connection connected from Ok(127.0.0.1:45987)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.234255526Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"io task","client":"1","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"ds_connection connected from Ok(127.0.0.1:48720)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.234291191Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"io task","client":"2","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.235440507Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"client":"1","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.23548747Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"client":"0","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.235992137Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.236077466Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.236318546Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"client":"2","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"[0]R flush_numbers[0..12]: [84, 82, 102, 95, 99, 98, 102, 79, 98, 95, 106, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.236793706Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"downstairs","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"[0]R generation[0..12]: [44, 42, 52, 49, 51, 50, 52, 41, 50, 49, 53, 52]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.236818373Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"downstairs","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.23683837Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"downstairs","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"[1]R flush_numbers[0..12]: [84, 82, 102, 95, 99, 98, 102, 79, 98, 95, 106, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.236860777Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"downstairs","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"[1]R generation[0..12]: [44, 42, 52, 49, 51, 50, 52, 41, 50, 49, 53, 52]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.236891294Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"downstairs","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.236925749Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"downstairs","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"[2]R flush_numbers[0..12]: [84, 82, 102, 95, 99, 98, 102, 79, 98, 95, 106, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.236956956Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"downstairs","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"[2]R generation[0..12]: [44, 42, 52, 49, 51, 50, 52, 41, 50, 49, 53, 52]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.236991071Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"downstairs","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.237014468Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"downstairs","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"Max found gen is 54","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.237031096Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"Generation requested: 54 >= found:54","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.237059863Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"downstairs","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"Next flush: 107","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.23707611Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"downstairs","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.237092598Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"":"downstairs","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.237109676Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.237125144Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"client":"0","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.237142002Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"client":"1","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.23715863Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"client":"2","":"downstairs","session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.237175578Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.237191686Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"931cb81c-6742-4b0f-a14f-1d4792ef2a02 is now active with session: 3ddf5b7c-216a-4969-97eb-d84883cdff3b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.237209424Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.237225222Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416,"session_id":"3ddf5b7c-216a-4969-97eb-d84883cdff3b"} {"msg":"The guest has finished waiting for activation with:54","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.23724106Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416} {"msg":"Activated sub_volume 931cb81c-6742-4b0f-a14f-1d4792ef2a02","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:18.237254758Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 302 len 16384 data: 4 4 6 7 02/30 Write block 216 len 32768 data: 3 3 2 2 3 3 3 4 03/30 Read block 59 len 16384 04/30 Flush 05/30 Write block 576 len 16384 data: 3 4 2 3 06/30 Write block 412 len 4096 data: 6 07/30 Read block 67 len 36864 08/30 Read block 399 len 4096 09/30 Read block 462 len 12288 10/30 Read block 461 len 36864 11/30 Read block 472 len 36864 12/30 Read block 187 len 12288 13/30 Read block 2 len 24576 14/30 Write block 461 len 8192 data: 2 3 15/30 Write block 192 len 4096 data: 3 16/30 Write block 331 len 12288 data: 4 6 6 17/30 Flush 18/30 Write block 265 len 36864 data: 3 5 6 5 6 7 8 8 6 19/30 Write block 31 len 16384 data: 4 4 5 7 20/30 Write block 381 len 4096 data: 4 21/30 Flush 22/30 Write block 531 len 16384 data: 5 5 7 6 23/30 Write block 534 len 24576 data: 7 6 6 5 4 3 24/30 Read block 120 len 28672 25/30 Read block 583 len 8192 26/30 Read block 412 len 20480 27/30 Read block 110 len 36864 28/30 Write block 535 len 28672 data: 7 7 6 5 4 4 6 29/30 Read block 7 len 28672 30/30 Read block 503 len 28672 ---------------------------------------------------------------- Crucible gen:54 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Read 4 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 4 Done Done Sent false 1005 Acked Write 1 Done Done Sent false 1006 Acked Read 9 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Read 3 Done Done Sent false 1009 Acked Read 9 Done Done Sent false 1010 Acked Read 9 Done Done Sent false 1011 Acked Read 3 Done Done Sent false 1012 Acked Read 6 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Write 1 Done Done Sent false 1015 Acked Write 3 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 9 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Write 1 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 4 Done Done Sent false 1022 Acked Write 6 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Read 2 Done Done Sent false 1025 Acked Read 5 Done Done Sent false 1026 Acked Read 9 Done Done Sent false 1027 Acked Write 7 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Read 7 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 28 28 Done 30 30 2 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:18.256122651Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1416} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 54 54 42 109 109 82 F F F <--- 2 040-059 52 52 52 102 102 102 F F F 3 060-079 49 49 49 95 95 95 F F F 4 080-099 51 51 51 99 99 99 F F F 5 100-119 50 50 50 98 98 98 F F F 6 120-139 52 52 52 102 102 102 F F F 7 140-159 41 41 41 79 79 79 F F F 8 160-179 50 50 50 98 98 98 F F F 9 180-199 54 54 49 108 108 95 F F F <--- 10 200-219 54 54 53 107 107 106 F F T <--- 11 220-239 54 54 52 107 107 105 F F T <--- 12 240-259 52 52 52 105 105 105 F F F 13 260-279 54 54 50 109 109 98 F F F <--- 14 280-299 53 53 53 106 106 106 F F F 15 300-319 54 54 49 107 107 95 F F T <--- 16 320-339 54 54 46 108 108 89 F F F <--- 17 340-359 53 53 53 106 106 106 F F F 18 360-379 53 53 53 106 106 106 F F F 19 380-399 54 54 48 109 109 94 F F F <--- 20 400-419 54 54 52 108 108 104 F F F <--- 21 420-439 52 52 52 103 103 103 F F F 22 440-459 52 52 52 103 103 103 F F F 23 460-479 54 54 49 108 108 95 F F F <--- 24 480-499 46 46 46 90 90 90 F F F 25 500-519 52 52 52 105 105 105 F F F 26 520-539 50 50 50 98 98 98 T T F <--- 27 540-559 52 52 52 103 103 103 T T F <--- 28 560-579 54 54 51 108 108 99 F F F <--- 29 580-599 51 51 51 99 99 99 F F F Max gen: 54, Max flush: 109 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 27 Verifying data now No change after verify Loop: 27 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 54 54 54 109 109 109 F F F 2 040-059 52 52 52 102 102 102 F F F 3 060-079 49 49 49 95 95 95 F F F 4 080-099 51 51 51 99 99 99 F F F 5 100-119 50 50 50 98 98 98 F F F 6 120-139 52 52 52 102 102 102 F F F 7 140-159 41 41 41 79 79 79 F F F 8 160-179 50 50 50 98 98 98 F F F 9 180-199 54 54 54 108 108 108 F F F 10 200-219 54 54 54 107 107 107 F F F 11 220-239 54 54 54 107 107 107 F F F 12 240-259 52 52 52 105 105 105 F F F 13 260-279 54 54 54 109 109 109 F F F 14 280-299 53 53 53 106 106 106 F F F 15 300-319 54 54 54 107 107 107 F F F 16 320-339 54 54 54 108 108 108 F F F 17 340-359 53 53 53 106 106 106 F F F 18 360-379 53 53 53 106 106 106 F F F 19 380-399 54 54 54 109 109 109 F F F 20 400-419 54 54 54 108 108 108 F F F 21 420-439 52 52 52 103 103 103 F F F 22 440-459 52 52 52 103 103 103 F F F 23 460-479 54 54 54 108 108 108 F F F 24 480-499 46 46 46 90 90 90 F F F 25 500-519 52 52 52 105 105 105 F F F 26 520-539 55 55 55 110 110 110 F F F 27 540-559 55 55 55 110 110 110 F F F 28 560-579 54 54 54 108 108 108 F F F 29 580-599 51 51 51 99 99 99 F F F Max gen: 55, Max flush: 110 Begin loop 28 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.66067141Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.705846788Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.70646345Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.706522883Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.70654839Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.706576706Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424} {"msg":"Crucible 76c9c4a0-d403-4ddb-bb42-cb5721e8a241 has session id: f08e6ba2-b98d-4358-8a88-403084ef021e","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.7070242Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"Upstairs opts: Upstairs UUID: 76c9c4a0-d403-4ddb-bb42-cb5721e8a241, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707080223Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"Crucible stats registered with UUID: 76c9c4a0-d403-4ddb-bb42-cb5721e8a241","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707108529Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:22.707133036Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707304305Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"1","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707345239Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"0","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707372716Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707525187Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"task":"crutest"} {"msg":"The guest has requested activation with gen:56","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707565161Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424} {"msg":"76c9c4a0-d403-4ddb-bb42-cb5721e8a241 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707598287Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707628893Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"client":"0","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707670098Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"client":"1","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707710203Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"client":"2","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707754148Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707799472Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"1","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.707842906Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"0","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:22.707890691Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:22.707938045Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"ds_connection connected from Ok(127.0.0.1:44591)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.7079788Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"1","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:22.708042112Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"ds_connection connected from Ok(127.0.0.1:44216)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.708079917Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"0","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.708140849Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.708185474Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.709220363Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"client":"1","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.709317781Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"client":"0","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.709859163Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:22.709898758Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.70940179Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"ds_connection connected from Ok(127.0.0.1:48208)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.709752476Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"io task","client":"2","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.711351955Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"client":"2","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"[0]R flush_numbers[0..12]: [84, 109, 102, 95, 99, 98, 102, 79, 98, 108, 107, 107]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.71194864Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"downstairs","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"[0]R generation[0..12]: [44, 54, 52, 49, 51, 50, 52, 41, 50, 54, 54, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.711984425Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"downstairs","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712010842Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"downstairs","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"[1]R flush_numbers[0..12]: [84, 109, 102, 95, 99, 98, 102, 79, 98, 108, 107, 107]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712046877Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"downstairs","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"[1]R generation[0..12]: [44, 54, 52, 49, 51, 50, 52, 41, 50, 54, 54, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712086042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"downstairs","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712131167Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"downstairs","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"[2]R flush_numbers[0..12]: [84, 109, 102, 95, 99, 98, 102, 79, 98, 108, 107, 107]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712159523Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"downstairs","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"[2]R generation[0..12]: [44, 54, 52, 49, 51, 50, 52, 41, 50, 54, 54, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712195398Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"downstairs","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712239733Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"downstairs","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"Max found gen is 56","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712271899Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"downstairs","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"Generation requested: 56 >= found:56","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712302075Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"downstairs","":"downstairsWait for a query_work_queue command to finish before sending IO ","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"Next flush: 111","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712350429Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"downstairs","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712388104Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"":"downstairs","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712415311Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712448457Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"client":"0","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712487202Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"client":"1","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712515118Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"client":"2","":"downstairs","session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712541015Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712568871Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"76c9c4a0-d403-4ddb-bb42-cb5721e8a241 is now active with session: f08e6ba2-b98d-4358-8a88-403084ef021e","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712598658Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712615576Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424,"session_id":"f08e6ba2-b98d-4358-8a88-403084ef021e"} {"msg":"The guest has finished waiting for activation with:56","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712641113Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Activated sub_volume 76c9c4a0-d403-4ddb-bb42-cb5721e8a241","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:32.712667879Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 205 len 4096 data: 8 02/30 Read block 157 len 12288 03/30 Write block 251 len 16384 data: 5 5 4 5 04/30 Write block 506 len 4096 data: 4 05/30 Write block 44 len 32768 data: 6 7 7 7 5 3 4 4 06/30 Read block 148 len 28672 07/30 Read block 264 len 4096 08/30 Read block 25 len 24576 09/30 Read block 403 len 24576 10/30 Write block 280 len 40960 data: 5 5 5 6 5 5 5 5 4 5 11/30 Write block 470 len 16384 data: 4 4 4 4 12/30 Read block 301 len 28672 13/30 Read block 68 len 24576 14/30 Write block 320 len 8192 data: 5 6 15/30 Write block 498 len 32768 data: 3 4 4 5 5 5 3 4 16/30 Flush 17/30 Write block 362 len 20480 data: 4 4 4 4 5 18/30 Read block 4 len 16384 19/30 Write block 535 len 24576 data: 8 8 7 6 5 5 20/30 Read block 374 len 32768 21/30 Read block 551 len 36864 22/30 Flush 23/30 Write block 430 len 20480 data: 7 5 4 4 7 24/30 Write block 159 len 4096 data: 5 25/30 Flush 26/30 Write block 539 len 40960 data: 6 6 7 8 6 6 7 7 8 9 27/30 Read block 275 len 12288 28/30 Write block 588 len 20480 data: 10 12 11 8 7 29/30 Write block 415 len 12288 data: 4 5 5 30/30 Write block 78 len 12288 data: 4 5 5 ---------------------------------------------------------------- Crucible gen:56 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Sent false 1001 Acked Read 3 Done Done Sent false 1002 Acked Write 4 Done Done Sent false 1003 Acked Write 1 Done Done Sent false 1004 Acked Write 8 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Read 1 Done Done Sent false 1007 Acked Read 6 Done Done Sent false 1008 Acked Read 6 Done Done Sent false 1009 Acked Write 10 Done Done Sent false 1010 Acked Write 4 Done Done Sent false 1011 Acked Read 7 Done Done Sent false 1012 Acked Read 6 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Write 8 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Read 4 Done Done Sent false 1018 Acked Write 6 Done Done Sent false 1019 Acked Read 8 Done Done Sent false 1020 Acked Read 9 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 5 Done Done Sent false 1023 Acked Write 1 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 10 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Write 5 Sent Sent Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 30 36 Done 27 27 0 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:32.731599344Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1424} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 54 54 54 109 109 109 F F F 2 040-059 56 56 52 111 111 102 F F F <--- 3 060-079 49 49 49 95 95 95 T T F <--- 4 080-099 51 51 51 99 99 99 T T F <--- 5 100-119 50 50 50 98 98 98 F F F 6 120-139 52 52 52 102 102 102 F F F 7 140-159 56 56 41 113 113 79 F F F <--- 8 160-179 50 50 50 98 98 98 F F F 9 180-199 54 54 54 108 108 108 F F F 10 200-219 56 56 54 111 111 107 F F F <--- 11 220-239 54 54 54 107 107 107 F F F 12 240-259 56 56 52 111 111 105 F F F <--- 13 260-279 54 54 54 109 109 109 F F F 14 280-299 56 56 53 111 111 106 F F F <--- 15 300-319 54 54 54 107 107 107 F F F 16 320-339 56 56 54 111 111 108 F F F <--- 17 340-359 53 53 53 106 106 106 F F F 18 360-379 56 56 53 112 112 106 F F F <--- 19 380-399 54 54 54 109 109 109 F F F 20 400-419 54 54 54 108 108 108 T T F <--- 21 420-439 56 56 52 113 113 103 F F F <--- 22 440-459 52 52 52 103 103 103 F F F 23 460-479 56 56 54 111 111 108 F F F <--- 24 480-499 56 56 46 111 111 90 F F F <--- 25 500-519 56 56 52 111 111 105 F F F <--- 26 520-539 56 56 55 112 112 110 T T F <--- 27 540-559 56 56 55 112 112 110 T T F <--- 28 560-579 54 54 54 108 108 108 F F F 29 580-599 51 51 51 99 99 99 T T F <--- Max gen: 56, Max flush: 113 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 28 Verifying data now No change after verify Loop: 28 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 54 54 54 109 109 109 F F F 2 040-059 56 56 56 111 111 111 F F F 3 060-079 57 57 57 114 114 114 F F F 4 080-099 57 57 57 114 114 114 F F F 5 100-119 50 50 50 98 98 98 F F F 6 120-139 52 52 52 102 102 102 F F F 7 140-159 56 56 56 113 113 113 F F F 8 160-179 50 50 50 98 98 98 F F F 9 180-199 54 54 54 108 108 108 F F F 10 200-219 56 56 56 111 111 111 F F F 11 220-239 54 54 54 107 107 107 F F F 12 240-259 56 56 56 111 111 111 F F F 13 260-279 54 54 54 109 109 109 F F F 14 280-299 56 56 56 111 111 111 F F F 15 300-319 54 54 54 107 107 107 F F F 16 320-339 56 56 56 111 111 111 F F F 17 340-359 53 53 53 106 106 106 F F F 18 360-379 56 56 56 112 112 112 F F F 19 380-399 54 54 54 109 109 109 F F F 20 400-419 57 57 57 114 114 114 F F F 21 420-439 56 56 56 113 113 113 F F F 22 440-459 52 52 52 103 103 103 F F F 23 460-479 56 56 56 111 111 111 F F F 24 480-499 56 56 56 111 111 111 F F F 25 500-519 56 56 56 111 111 111 F F F 26 520-539 57 57 57 114 114 114 F F F 27 540-559 57 57 57 114 114 114 F F F 28 560-579 54 54 54 108 108 108 F F F 29 580-599 57 57 57 114 114 114 F F F Max gen: 57, Max flush: 114 Begin loop 29 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.157993901Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.203369334Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.204100922Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.204151436Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.204168794Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.20419518Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432} {"msg":"Crucible 65af3788-ba61-492c-acfb-968493cb19a4 has session id: a63fb84f-3c1d-4f18-9636-432d13f185e2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.204773417Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"Upstairs opts: Upstairs UUID: 65af3788-ba61-492c-acfb-968493cb19a4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.204845708Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"Crucible stats registered with UUID: 65af3788-ba61-492c-acfb-968493cb19a4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.204875154Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:37.204896232Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205106196Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"io task","client":"2","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205154549Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"io task","client":"0","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205184946Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"io task","client":"1","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205302981Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"task":"crutest"} {"msg":"The guest has requested activation with gen:58","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205329387Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432} {"msg":"65af3788-ba61-492c-acfb-968493cb19a4 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205397599Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205418026Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"client":"0","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205436964Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"client":"1","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.20546543Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"client":"2","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205496646Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"io task","client":"2","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205536001Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"io task","client":"0","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205571707Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"io task","client":"1","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"ds_connection connected from Ok(127.0.0.1:55348)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205607002Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"io task","client":"2","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"ds_connection connected from Ok(127.0.0.1:56919)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205639048Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"io task","client":"1","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"ds_connection connected from Ok(127.0.0.1:50878)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.205670784Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"io task","client":"0","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207047431Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"client":"1","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207084106Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"client":"0","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207121152Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"client":"2","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207602581Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.20769227Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"[0]R flush_numbers[0..12]: [84, 109, 111, 114, 114, 98, 102, 113, 98, 108, 111, 107]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207776999Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"downstairs","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"[0]R generation[0..12]: [44, 54, 56, 57, 57, 50, 52, 56, 50, 54, 56, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207801026Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"downstairs","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207821393Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"downstairs","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"[1]R flush_numbers[0..12]: [84, 109, 111, 114, 114, 98, 102, 113, 98, 108, 111, 107]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207840711Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"downstairs","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"[1]R generation[0..12]: [44, 54, 56, 57, 57, 50, 52, 56, 50, 54, 56, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207862498Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"downstairs","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207882396Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"downstairs","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"[2]R flush_numbers[0..12]: [84, 109, 111, 114, 114, 98, 102, 113, 98, 108, 111, 107]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207901323Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"downstairs","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"[2]R generation[0..12]: [44, 54, 56, 57, 57, 50, 52, 56, 50, 54, 56, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207919591Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"downstairs","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207939068Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"downstairs","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"Max found gen is 58","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207955477Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"downstairs","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"Generation requested: 58 >= found:58","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-11-06T20:13:37.207972474Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"downstairs","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"Next flush: 115","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.207992762Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"downstairs","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.208009Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"":"downstairs","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.208026088Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.208041596Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"client":"0","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.208058444Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"client":"1","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.208075181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"client":"2","":"downstairs","session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.208092249Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.208108277Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"65af3788-ba61-492c-acfb-968493cb19a4 is now active with session: a63fb84f-3c1d-4f18-9636-432d13f185e2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.208127765Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.208142983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432,"session_id":"a63fb84f-3c1d-4f18-9636-432d13f185e2"} {"msg":"The guest has finished waiting for activation with:58","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:13:37.208158971Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432} {"msg":"Activated sub_volume 65af3788-ba61-492c-acfb-968493cb19a4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:37.208176509Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 176 len 12288 data: 5 5 5 02/30 Write block 342 len 12288 data: 4 4 4 03/30 Read block 328 len 20480 04/30 Read block 354 len 12288 05/30 Read block 495 len 16384 06/30 Write block 332 len 32768 data: 7 7 6 5 5 5 6 6 07/30 Read block 404 len 20480 08/30 Read block 502 len 20480 09/30 Read block 317 len 32768 10/30 Read block 430 len 40960 11/30 Write block 29 len 8192 data: 4 4 12/30 Write block 399 len 36864 data: 5 5 7 6 5 5 5 4 3 13/30 Write block 164 len 24576 data: 3 4 4 4 5 5 14/30 Read block 56 len 20480 15/30 Read block 587 len 40960 16/30 Read block 446 len 16384 17/30 Write block 450 len 40960 data: 7 9 8 8 10 11 10 8 6 6 18/30 Read block 120 len 24576 19/30 Read block 550 len 28672 20/30 Read block 548 len 24576 21/30 Read block 72 len 8192 22/30 Read block 446 len 28672 23/30 Read block 176 len 20480 24/30 Read block 274 len 12288 25/30 Write block 99 len 40960 data: 5 4 3 4 3 4 3 3 4 4 26/30 Write block 40 len 8192 data: 8 7 27/30 Write block 116 len 8192 data: 15 12 28/30 Read block 375 len 4096 29/30 Read block 576 len 32768 30/30 Read block 567 len 16384 ---------------------------------------------------------------- Crucible gen:58 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 3 Done Done Done false 1002 Acked Read 5 Done Done Done false 1003 Acked Read 3 Done Done Done false 1004 Acked Read 4 Done Done Done false 1005 Acked Write 8 Done Done Sent false 1006 Acked Read 5 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Read 8 Done Done Sent false 1009 Acked Read 10 Done Done Sent false 1010 Acked Write 2 Done Done Sent false 1011 Acked Write 9 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Read 10 Done Done Sent false 1015 Acked Read 4 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Read 5 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Write 2 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 1 Done Done Sent false 1028 Acked Read 8 Done Done Sent false 1029 Acked Read 4 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 25 25 Done 30 30 5 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:37.227635977Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1432} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 54 54 54 109 109 109 T T F <--- 2 040-059 56 56 56 111 111 111 T T F <--- 3 060-079 57 57 57 114 114 114 F F F 4 080-099 57 57 57 114 114 114 T T F <--- 5 100-119 50 50 50 98 98 98 T T F <--- 6 120-139 52 52 52 102 102 102 F F F 7 140-159 56 56 56 113 113 113 F F F 8 160-179 50 50 50 98 98 98 T T T 9 180-199 54 54 54 108 108 108 F F F 10 200-219 56 56 56 111 111 111 F F F 11 220-239 54 54 54 107 107 107 F F F 12 240-259 56 56 56 111 111 111 F F F 13 260-279 54 54 54 109 109 109 F F F 14 280-299 56 56 56 111 111 111 F F F 15 300-319 54 54 54 107 107 107 F F F 16 320-339 56 56 56 111 111 111 T T F <--- 17 340-359 53 53 53 106 106 106 T T T 18 360-379 56 56 56 112 112 112 F F F 19 380-399 54 54 54 109 109 109 T T F <--- 20 400-419 57 57 57 114 114 114 T T F <--- 21 420-439 56 56 56 113 113 113 F F F 22 440-459 52 52 52 103 103 103 T T F <--- 23 460-479 56 56 56 111 111 111 F F F 24 480-499 56 56 56 111 111 111 F F F 25 500-519 56 56 56 111 111 111 F F F 26 520-539 57 57 57 114 114 114 F F F 27 540-559 57 57 57 114 114 114 F F F 28 560-579 54 54 54 108 108 108 F F F 29 580-599 57 57 57 114 114 114 F F F Max gen: 57, Max flush: 114 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 29 Verifying data now No change after verify Loop: 29 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 58 58 58 115 115 115 F F F 2 040-059 58 58 58 115 115 115 F F F 3 060-079 57 57 57 114 114 114 F F F 4 080-099 58 58 58 115 115 115 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 52 52 52 102 102 102 F F F 7 140-159 56 56 56 113 113 113 F F F 8 160-179 58 58 58 115 115 115 F F F 9 180-199 54 54 54 108 108 108 F F F 10 200-219 56 56 56 111 111 111 F F F 11 220-239 54 54 54 107 107 107 F F F 12 240-259 56 56 56 111 111 111 F F F 13 260-279 54 54 54 109 109 109 F F F 14 280-299 56 56 56 111 111 111 F F F 15 300-319 54 54 54 107 107 107 F F F 16 320-339 58 58 58 115 115 115 F F F 17 340-359 58 58 58 115 115 115 F F F 18 360-379 56 56 56 112 112 112 F F F 19 380-399 58 58 58 115 115 115 F F F 20 400-419 58 58 58 115 115 115 F F F 21 420-439 56 56 56 113 113 113 F F F 22 440-459 58 58 58 115 115 115 F F F 23 460-479 56 56 56 111 111 111 F F F 24 480-499 56 56 56 111 111 111 F F F 25 500-519 56 56 56 111 111 111 F F F 26 520-539 57 57 57 114 114 114 F F F 27 540-559 57 57 57 114 114 114 F F F 28 560-579 54 54 54 108 108 108 F F F 29 580-599 57 57 57 114 114 114 F F F Max gen: 58, Max flush: 115 Begin loop 30 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.625051515Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.6712088Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.671867037Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.67192316Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.671942748Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.671970124Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440} {"msg":"Crucible e3d77e49-5b1c-4c94-9ee4-6834eb07a6dd has session id: 46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.672530514Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"Upstairs opts: Upstairs UUID: e3d77e49-5b1c-4c94-9ee4-6834eb07a6dd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.672589016Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"Crucible stats registered with UUID: e3d77e49-5b1c-4c94-9ee4-6834eb07a6dd","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.672620882Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:41.672651668Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.672877Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.672914385Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"io task","client":"2","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.672938292Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"io task","client":"0","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.673091133Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"task":"crutest"} {"msg":"The guest has requested activation with gen:60","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.67311769Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440} {"msg":"e3d77e49-5b1c-4c94-9ee4-6834eb07a6dd active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.673185701Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.673206229Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"client":"0","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.673229646Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"client":"1","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.673259072Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"client":"2","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.673293627Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"io task","client":"2","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.673327983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"io task","client":"0","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.673362229Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"ds_connection connected from Ok(127.0.0.1:37337)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.673397694Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"io task","client":"0","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"ds_connection connected from Ok(127.0.0.1:51533)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.6734326Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"io task","client":"1","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"ds_connection connected from Ok(127.0.0.1:33414)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.673474615Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"io task","client":"2","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.67470473Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"client":"0","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.674814466Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"client":"1","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.674840243Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"client":"2","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675290626Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675383414Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"[0]R flush_numbers[0..12]: [84, 115, 115, 114, 115, 115, 102, 113, 115, 108, 111, 107]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.67549374Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"downstairs","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"[0]R generation[0..12]: [44, 58, 58, 57, 58, 58, 52, 56, 58, 54, 56, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675518947Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"downstairs","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675539804Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"downstairs","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"[1]R flush_numbers[0..12]: [84, 115, 115, 114, 115, 115, 102, 113, 115, 108, 111, 107]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675558942Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"downstairs","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"[1]R generation[0..12]: [44, 58, 58, 57, 58, 58, 52, 56, 58, 54, 56, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.67557755Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"downstairs","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675597427Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"downstairs","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"[2]R flush_numbers[0..12]: [84, 115, 115, 114, 115, 115, 102, 113, 115, 108, 111, 107]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675616345Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"downstairs","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"[2]R generation[0..12]: [44, 58, 58, 57, 58, 58, 52, 56, 58, 54, 56, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675634743Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"downstairs","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.67565428Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"downstairs","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"Max found gen is 59","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675670408Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"downstairs","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"Generation requested: 60 >= found:59","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-11-06T20:13:41.675687126Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"downstairs","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"Next flush: 116","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675709943Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"downstairs","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675729081Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"":"downstairs","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675746078Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675761407Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"client":"0","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675778354Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"client":"1","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675794882Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"client":"2","":"downstairs","session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.67581173Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675827818Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"e3d77e49-5b1c-4c94-9ee4-6834eb07a6dd is now active with session: 46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675845636Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675863834Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440,"session_id":"46a5bbd5-bff0-44b1-ab46-c7b8f5a8aaa6"} {"msg":"The guest has finished waiting for activation with:60","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.675880062Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440} {"msg":"Activated sub_volume e3d77e49-5b1c-4c94-9ee4-6834eb07a6dd","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:41.67589389Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 201 len 4096 data: 5 02/30 Write block 240 len 36864 data: 6 7 7 7 7 6 5 3 3 03/30 Write block 38 len 24576 data: 9 8 9 8 6 5 04/30 Read block 490 len 4096 05/30 Write block 456 len 24576 data: 11 9 7 7 3 3 06/30 Write block 299 len 24576 data: 4 4 4 5 5 7 07/30 Write block 181 len 24576 data: 10 10 10 5 5 6 08/30 Flush 09/30 Write block 436 len 16384 data: 6 5 5 8 10/30 Read block 393 len 28672 11/30 Flush 12/30 Write block 508 len 40960 data: 5 5 6 6 6 6 5 5 5 8 13/30 Read block 550 len 16384 14/30 Read block 528 len 4096 15/30 Read block 283 len 32768 16/30 Read block 214 len 36864 17/30 Read block 439 len 8192 18/30 Write block 136 len 28672 data: 8 7 7 6 7 7 8 19/30 Read block 362 len 28672 20/30 Flush 21/30 Write block 407 len 28672 data: 4 5 6 7 7 7 6 22/30 Write block 266 len 32768 data: 6 7 6 7 8 9 9 7 23/30 Flush 24/30 Write block 554 len 28672 data: 8 7 6 5 4 3 2 25/30 Write block 321 len 32768 data: 7 5 6 6 7 7 6 5 26/30 Read block 378 len 36864 27/30 Write block 85 len 28672 data: 4 5 5 6 8 7 6 28/30 Read block 197 len 8192 29/30 Read block 312 len 12288 30/30 Read block 462 len 8192 ---------------------------------------------------------------- Crucible gen:60 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 9 Done Done Sent false 1002 Acked Write 6 Done Done Sent false 1003 Acked Read 1 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Write 6 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Read 7 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Read 1 Done Done Sent false 1014 Acked Read 8 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Write 7 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 7 Done Done Sent false 1021 Acked Write 8 Done Done Sent false 1022 Acked Flush 0 Done Done Sent false 1023 Acked Write 7 Done Done Sent false 1024 Acked Write 8 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Read 3 Done Done Sent false 1029 Acked Read 2 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 29 29 Done 30 30 1 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) Some(JobId(1022)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:41.695395603Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1440} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 60 60 58 116 116 115 F F F <--- 2 040-059 60 60 58 116 116 115 F F F <--- 3 060-079 57 57 57 114 114 114 F F F 4 080-099 58 58 58 115 115 115 T T F <--- 5 100-119 58 58 58 115 115 115 F F F 6 120-139 60 60 52 118 118 102 F F F <--- 7 140-159 60 60 56 118 118 113 F F F <--- 8 160-179 58 58 58 115 115 115 F F F 9 180-199 60 60 54 116 116 108 F F F <--- 10 200-219 60 60 56 116 116 111 F F T <--- 11 220-239 54 54 54 107 107 107 F F F 12 240-259 60 60 56 116 116 111 F F F <--- 13 260-279 60 60 54 119 119 109 F F F <--- 14 280-299 60 60 56 116 116 111 F F F <--- 15 300-319 60 60 54 116 116 107 F F F <--- 16 320-339 58 58 58 115 115 115 T T F <--- 17 340-359 58 58 58 115 115 115 F F F 18 360-379 56 56 56 112 112 112 F F F 19 380-399 58 58 58 115 115 115 F F F 20 400-419 60 60 58 119 119 115 F F F <--- 21 420-439 60 60 56 117 117 113 F F F <--- 22 440-459 60 60 58 116 116 115 F F F <--- 23 460-479 60 60 56 116 116 111 F F F <--- 24 480-499 56 56 56 111 111 111 F F F 25 500-519 60 60 56 118 118 111 F F F <--- 26 520-539 57 57 57 114 114 114 F F F 27 540-559 57 57 57 114 114 114 T T F <--- 28 560-579 54 54 54 108 108 108 T T F <--- 29 580-599 57 57 57 114 114 114 F F F Max gen: 60, Max flush: 119 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 30 Verifying data now No change after verify Loop: 30 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 60 60 60 116 116 116 F F F 2 040-059 60 60 60 116 116 116 F F F 3 060-079 57 57 57 114 114 114 F F F 4 080-099 61 61 61 120 120 120 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 60 60 60 118 118 118 F F F 7 140-159 60 60 60 118 118 118 F F F 8 160-179 58 58 58 115 115 115 F F F 9 180-199 60 60 60 116 116 116 F F F 10 200-219 60 60 60 116 116 116 F F F 11 220-239 54 54 54 107 107 107 F F F 12 240-259 60 60 60 116 116 116 F F F 13 260-279 60 60 60 119 119 119 F F F 14 280-299 60 60 60 116 116 116 F F F 15 300-319 60 60 60 116 116 116 F F F 16 320-339 61 61 61 120 120 120 F F F 17 340-359 58 58 58 115 115 115 F F F 18 360-379 56 56 56 112 112 112 F F F 19 380-399 58 58 58 115 115 115 F F F 20 400-419 60 60 60 119 119 119 F F F 21 420-439 60 60 60 117 117 117 F F F 22 440-459 60 60 60 116 116 116 F F F 23 460-479 60 60 60 116 116 116 F F F 24 480-499 56 56 56 111 111 111 F F F 25 500-519 60 60 60 118 118 118 F F F 26 520-539 57 57 57 114 114 114 F F F 27 540-559 61 61 61 120 120 120 F F F 28 560-579 61 61 61 120 120 120 F F F 29 580-599 57 57 57 114 114 114 F F F Max gen: 61, Max flush: 120 Begin loop 31 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.127278058Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:46.171168798Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.171570588Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.203120603Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.20369731Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.203762902Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.203787898Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.203815845Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448} {"msg":"Crucible 74721115-5eb8-4f23-a486-efe769ca2d6f has session id: 3abbc9b6-9272-4c9f-940c-304925e07cb6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.204315682Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"Upstairs opts: Upstairs UUID: 74721115-5eb8-4f23-a486-efe769ca2d6f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.204458654Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"Crucible stats registered with UUID: 74721115-5eb8-4f23-a486-efe769ca2d6f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.20449146Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:46.204509428Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.204738189Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"io task","client":"0","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.204774934Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"io task","client":"2","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.204794652Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"io task","client":"1","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.204983758Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"task":"crutest"} {"msg":"The guest has requested activation with gen:62","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.205018943Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448} {"msg":"74721115-5eb8-4f23-a486-efe769ca2d6f active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.205083715Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.205104503Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"client":"0","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.205121661Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"client":"1","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.205141598Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"client":"2","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.205158326Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"io task","client":"2","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.20520451Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"io task","client":"0","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.205266772Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"io task","client":"1","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"ds_connection connected from Ok(127.0.0.1:52677)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.205305557Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"io task","client":"1","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"ds_connection connected from Ok(127.0.0.1:58369)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.205338293Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"io task","client":"0","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"ds_connection connected from Ok(127.0.0.1:49840)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.205369549Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"io task","client":"2","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.206542862Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"client":"1","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.206576127Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"client":"0","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.20663645Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"client":"2","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207155454Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.20727311Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"[0]R flush_numbers[0..12]: [84, 116, 116, 114, 120, 115, 118, 118, 115, 116, 116, 107]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207311465Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"downstairs","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"[0]R generation[0..12]: [44, 60, 60, 57, 61, 58, 60, 60, 58, 60, 60, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207332172Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"downstairs","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207352409Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"downstairs","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"[1]R flush_numbers[0..12]: [84, 116, 116, 114, 120, 115, 118, 118, 115, 116, 116, 107]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207374657Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"downstairs","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"[1]R generation[0..12]: [44, 60, 60, 57, 61, 58, 60, 60, 58, 60, 60, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207393294Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"downstairs","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207413122Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"downstairs","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"[2]R flush_numbers[0..12]: [84, 116, 116, 114, 120, 115, 118, 118, 115, 116, 116, 107]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207431999Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"downstairs","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"[2]R generation[0..12]: [44, 60, 60, 57, 61, 58, 60, 60, 58, 60, 60, 54]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207450177Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"downstairs","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207469765Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"downstairs","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"Max found gen is 62","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207486043Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"downstairs","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"Generation requested: 62 >= found:62","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207502961Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"downstairs","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"Next flush: 121","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207519059Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"downstairs","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"All extents match","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-11-06T20:13:46.207542925Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"":"downstairs","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207566702Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.20758216Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"client":"0","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207599188Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"client":"1","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207615866Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"client":"2","":"downstairs","session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207632934Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207651092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"74721115-5eb8-4f23-a486-efe769ca2d6f is now active with session: 3abbc9b6-9272-4c9f-940c-304925e07cb6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.2076693Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207684538Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448,"session_id":"3abbc9b6-9272-4c9f-940c-304925e07cb6"} {"msg":"The guest has finished waiting for activation with:62","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207700376Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448} {"msg":"Activated sub_volume 74721115-5eb8-4f23-a486-efe769ca2d6f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:46.207714074Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 472 len 8192 data: 5 5 02/30 Read block 19 len 8192 03/30 Write block 363 len 28672 data: 5 5 5 6 4 4 3 04/30 Read block 94 len 12288 05/30 Write block 498 len 24576 data: 4 5 5 6 6 6 06/30 Write block 566 len 12288 data: 4 5 3 07/30 Write block 218 len 24576 data: 3 3 4 4 4 5 08/30 Write block 351 len 40960 data: 4 3 4 3 5 5 4 3 3 2 09/30 Read block 200 len 12288 10/30 Read block 315 len 28672 11/30 Read block 509 len 16384 12/30 Read block 140 len 4096 13/30 Write block 426 len 4096 data: 8 14/30 Write block 125 len 20480 data: 5 5 6 9 11 15/30 Read block 224 len 8192 16/30 Write block 503 len 16384 data: 7 4 5 5 17/30 Flush 18/30 Write block 155 len 4096 data: 4 19/30 Read block 433 len 20480 20/30 Write block 353 len 36864 data: 5 4 6 6 5 4 4 3 4 21/30 Read block 381 len 8192 22/30 Flush 23/30 Write block 483 len 12288 data: 5 5 5 24/30 Write block 192 len 40960 data: 4 3 3 2 3 4 4 4 4 6 25/30 Read block 355 len 16384 26/30 Write block 74 len 20480 data: 3 3 4 4 5 27/30 Write block 542 len 12288 data: 9 7 7 28/30 Read block 31 len 20480 29/30 Read block 297 len 12288 30/30 Write block 49 len 8192 data: 4 5 ---------------------------------------------------------------- Crucible gen:62 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Sent Done Done false 1001 Acked Read 2 Sent Done Done false 1002 Acked Write 7 Sent Done Done false 1003 Acked Read 3 Sent Done Done false 1004 Acked Write 6 Sent Done Done false 1005 Acked Write 3 Sent Done Done false 1006 Acked Write 6 Sent Done Done false 1007 Acked Write 10 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Read 1 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Write 4 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Read 5 Sent Done Done false 1019 Acked Write 9 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Write 5 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:46.225875156Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1448} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 60 60 60 116 116 116 F F F 2 040-059 60 60 60 116 116 116 F T T <--- 3 060-079 57 57 57 114 114 114 F T T <--- 4 080-099 61 61 61 120 120 120 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 60 62 62 118 121 121 F F F <--- 7 140-159 60 62 62 118 122 122 F F F <--- 8 160-179 58 58 58 115 115 115 F F F 9 180-199 60 60 60 116 116 116 F T T <--- 10 200-219 60 62 62 116 121 121 F T T <--- 11 220-239 54 62 62 107 121 121 F F F <--- 12 240-259 60 60 60 116 116 116 F F F 13 260-279 60 60 60 119 119 119 F F F 14 280-299 60 60 60 116 116 116 F F F 15 300-319 60 60 60 116 116 116 F F F 16 320-339 61 61 61 120 120 120 F F F 17 340-359 58 62 62 115 122 122 F F F <--- 18 360-379 56 62 62 112 122 122 F F F <--- 19 380-399 58 58 58 115 115 115 F F F 20 400-419 60 60 60 119 119 119 F F F 21 420-439 60 62 62 117 121 121 F F F <--- 22 440-459 60 60 60 116 116 116 F F F 23 460-479 60 62 62 116 121 121 F F F <--- 24 480-499 56 62 62 111 121 121 F T T <--- 25 500-519 60 62 62 118 121 121 F F F <--- 26 520-539 57 57 57 114 114 114 F F F 27 540-559 61 61 61 120 120 120 F T T <--- 28 560-579 61 62 62 120 121 121 F F F <--- 29 580-599 57 57 57 114 114 114 F F F Max gen: 62, Max flush: 122 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 31 Verifying data now No change after verify Loop: 31 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 60 60 60 116 116 116 F F F 2 040-059 63 63 63 123 123 123 F F F 3 060-079 63 63 63 123 123 123 F F F 4 080-099 61 61 61 120 120 120 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 62 62 62 121 121 121 F F F 7 140-159 62 62 62 122 122 122 F F F 8 160-179 58 58 58 115 115 115 F F F 9 180-199 63 63 63 123 123 123 F F F 10 200-219 63 63 63 123 123 123 F F F 11 220-239 62 62 62 121 121 121 F F F 12 240-259 60 60 60 116 116 116 F F F 13 260-279 60 60 60 119 119 119 F F F 14 280-299 60 60 60 116 116 116 F F F 15 300-319 60 60 60 116 116 116 F F F 16 320-339 61 61 61 120 120 120 F F F 17 340-359 62 62 62 122 122 122 F F F 18 360-379 62 62 62 122 122 122 F F F 19 380-399 58 58 58 115 115 115 F F F 20 400-419 60 60 60 119 119 119 F F F 21 420-439 62 62 62 121 121 121 F F F 22 440-459 60 60 60 116 116 116 F F F 23 460-479 62 62 62 121 121 121 F F F 24 480-499 63 63 63 123 123 123 F F F 25 500-519 62 62 62 121 121 121 F F F 26 520-539 57 57 57 114 114 114 F F F 27 540-559 63 63 63 123 123 123 F F F 28 560-579 62 62 62 121 121 121 F F F 29 580-599 57 57 57 114 114 114 F F F Max gen: 63, Max flush: 123 Begin loop 32 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.669730803Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.715147352Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.71579364Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.715847223Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.715863741Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.715886248Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456} {"msg":"Crucible 74bdfa6d-944a-4431-ae51-ed9d0a23e5e5 has session id: b0c7e56a-8dc8-44d4-bff9-b885be104528","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.716495692Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"Upstairs opts: Upstairs UUID: 74bdfa6d-944a-4431-ae51-ed9d0a23e5e5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.716545785Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"Crucible stats registered with UUID: 74bdfa6d-944a-4431-ae51-ed9d0a23e5e5","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.716566063Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:50.716582541Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.716797904Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.716833919Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.716853557Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.717088967Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"task":"crutest"} {"msg":"The guest has requested activation with gen:64","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.717116833Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456} {"msg":"74bdfa6d-944a-4431-ae51-ed9d0a23e5e5 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.717176356Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.717204532Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"client":"0","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.71722248Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"client":"1","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.717239398Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"client":"2","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.717256026Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.717273364Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.7173011Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"ds_connection connected from Ok(127.0.0.1:38101)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.717410866Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"io task","client":"0","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"ds_connection connected from Ok(127.0.0.1:54869)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.717471359Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"io task","client":"2","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"ds_connection connected from Ok(127.0.0.1:32945)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.717554838Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"io task","client":"1","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.718592798Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"client":"0","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.718762636Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"client":"2","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.719173214Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.719256614Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"client":"1","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.719373339Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"[0]R flush_numbers[0..12]: [84, 116, 123, 123, 120, 115, 121, 122, 115, 123, 123, 121]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.719775869Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"downstairs","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"[0]R generation[0..12]: [44, 60, 63, 63, 61, 58, 62, 62, 58, 63, 63, 62]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.719809494Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"downstairs","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.719830862Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"downstairs","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"[1]R flush_numbers[0..12]: [84, 116, 123, 123, 120, 115, 121, 122, 115, 123, 123, 121]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.719850449Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"downstairs","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"[1]R generation[0..12]: [44, 60, 63, 63, 61, 58, 62, 62, 58, 63, 63, 62]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.719869217Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"downstairs","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.719888784Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"downstairs","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"[2]R flush_numbers[0..12]: [84, 116, 123, 123, 120, 115, 121, 122, 115, 123, 123, 121]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.719907632Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"downstairs","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"[2]R generation[0..12]: [44, 60, 63, 63, 61, 58, 62, 62, 58, 63, 63, 62]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.719925799Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"downstairs","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.719955896Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"downstairs","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"Max found gen is 64","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.720010799Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"downstairs","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"Generation requested: 64 >= found:64","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.720042615Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"downstairs","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"Next flush: 124","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.720072121Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"downstairs","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"All extents match","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:13:50.720101337Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"":"downstairs","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.720145152Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.7201635Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"client":"0","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.720180607Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"client":"1","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.720197675Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"client":"2","":"downstairs","session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.720214923Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.720231731Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"74bdfa6d-944a-4431-ae51-ed9d0a23e5e5 is now active with session: b0c7e56a-8dc8-44d4-bff9-b885be104528","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.720249749Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.720265037Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456,"session_id":"b0c7e56a-8dc8-44d4-bff9-b885be104528"} {"msg":"The guest has finished waiting for activation with:64","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.720281045Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456} {"msg":"Activated sub_volume 74bdfa6d-944a-4431-ae51-ed9d0a23e5e5","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:50.720294923Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 127 len 32768 data: 7 10 12 9 9 9 8 7 02/30 Read block 146 len 36864 03/30 Write block 253 len 16384 data: 5 6 5 5 04/30 Read block 529 len 40960 05/30 Write block 47 len 36864 data: 8 6 5 6 5 3 4 4 3 06/30 Read block 138 len 20480 07/30 Write block 585 len 4096 data: 6 08/30 Write block 205 len 8192 data: 9 7 09/30 Flush 10/30 Write block 213 len 32768 data: 3 3 3 4 4 4 4 5 11/30 Read block 414 len 8192 12/30 Read block 407 len 28672 13/30 Write block 42 len 40960 data: 7 6 7 8 8 9 7 6 7 6 14/30 Write block 120 len 4096 data: 10 15/30 Write block 313 len 28672 data: 7 7 5 5 5 4 4 16/30 Write block 177 len 36864 data: 6 6 9 9 11 11 11 6 6 17/30 Read block 562 len 4096 18/30 Read block 96 len 32768 19/30 Write block 219 len 24576 data: 5 6 5 5 6 3 20/30 Write block 78 len 16384 data: 6 6 6 5 21/30 Read block 407 len 32768 22/30 Read block 138 len 16384 23/30 Write block 267 len 36864 data: 8 7 8 9 10 10 8 4 3 24/30 Read block 257 len 12288 25/30 Write block 443 len 36864 data: 7 8 7 4 4 6 7 8 10 26/30 Write block 95 len 16384 data: 6 6 6 4 27/30 Read block 466 len 8192 28/30 Read block 527 len 24576 29/30 Read block 459 len 36864 30/30 Write block 197 len 8192 data: 5 5 ---------------------------------------------------------------- Crucible gen:64 GIO:true work queues: Upstairs:0 downstairs:21 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1009 Acked Write 8 Done Done Done false 1010 Acked Read 2 Done Done Done false 1011 Acked Read 7 Done Done Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Read 8 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Read 4 Done Sent Done false 1022 Acked Write 9 Done Sent Done false 1023 Acked Read 3 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Write 4 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 18 1 20 Done 20 3 20 43 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1008)) Some(JobId(1008)) Some(JobId(1008)) Downstairs last five completed: 1008 1007 1006 1005 1004 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:50.738983478Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1456} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 60 60 60 116 116 116 F F F 2 040-059 64 64 64 124 124 124 T F T <--- 3 060-079 63 63 63 123 123 123 T F T <--- 4 080-099 61 61 61 120 120 120 T F T <--- 5 100-119 58 58 58 115 115 115 F F F 6 120-139 64 64 64 124 124 124 T F T <--- 7 140-159 62 62 62 122 122 122 F F F 8 160-179 58 58 58 115 115 115 T F T <--- 9 180-199 63 63 63 123 123 123 T F T <--- 10 200-219 64 64 64 124 124 124 T T T 11 220-239 62 62 62 121 121 121 T T T 12 240-259 64 64 64 124 124 124 F F F 13 260-279 60 60 60 119 119 119 T F T <--- 14 280-299 60 60 60 116 116 116 F F F 15 300-319 60 60 60 116 116 116 T F T <--- 16 320-339 61 61 61 120 120 120 F F F 17 340-359 62 62 62 122 122 122 F F F 18 360-379 62 62 62 122 122 122 F F F 19 380-399 58 58 58 115 115 115 F F F 20 400-419 60 60 60 119 119 119 F F F 21 420-439 62 62 62 121 121 121 F F F 22 440-459 60 60 60 116 116 116 T F T <--- 23 460-479 62 62 62 121 121 121 F F F 24 480-499 63 63 63 123 123 123 F F F 25 500-519 62 62 62 121 121 121 F F F 26 520-539 57 57 57 114 114 114 F F F 27 540-559 63 63 63 123 123 123 F F F 28 560-579 62 62 62 121 121 121 F F F 29 580-599 64 64 64 124 124 124 F F F Max gen: 64, Max flush: 124 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 32 Verifying data now No change after verify Loop: 32 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 60 60 60 116 116 116 F F F 2 040-059 65 65 65 125 125 125 F F F 3 060-079 65 65 65 125 125 125 F F F 4 080-099 65 65 65 125 125 125 F F F 5 100-119 58 58 58 115 115 115 F F F 6 120-139 65 65 65 125 125 125 F F F 7 140-159 62 62 62 122 122 122 F F F 8 160-179 65 65 65 125 125 125 F F F 9 180-199 65 65 65 125 125 125 F F F 10 200-219 65 65 65 125 125 125 F F F 11 220-239 65 65 65 125 125 125 F F F 12 240-259 64 64 64 124 124 124 F F F 13 260-279 65 65 65 125 125 125 F F F 14 280-299 60 60 60 116 116 116 F F F 15 300-319 65 65 65 125 125 125 F F F 16 320-339 61 61 61 120 120 120 F F F 17 340-359 62 62 62 122 122 122 F F F 18 360-379 62 62 62 122 122 122 F F F 19 380-399 58 58 58 115 115 115 F F F 20 400-419 60 60 60 119 119 119 F F F 21 420-439 62 62 62 121 121 121 F F F 22 440-459 65 65 65 125 125 125 F F F 23 460-479 62 62 62 121 121 121 F F F 24 480-499 63 63 63 123 123 123 F F F 25 500-519 62 62 62 121 121 121 F F F 26 520-539 57 57 57 114 114 114 F F F 27 540-559 63 63 63 123 123 123 F F F 28 560-579 62 62 62 121 121 121 F F F 29 580-599 64 64 64 124 124 124 F F F Max gen: 65, Max flush: 125 Begin loop 33 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.14137568Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.186548379Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.187244921Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.187294215Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.187324421Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.187347478Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464} {"msg":"Crucible 417760d2-a386-483a-b03b-119b2bbbe63c has session id: 1a3ffe30-1d75-4759-902d-4793f35e4c44","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.187947962Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"Upstairs opts: Upstairs UUID: 417760d2-a386-483a-b03b-119b2bbbe63c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188004965Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"Crucible stats registered with UUID: 417760d2-a386-483a-b03b-119b2bbbe63c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188038691Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:55.188066177Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188305847Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"io task","client":"0","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188344912Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"io task","client":"1","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.18836426Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"io task","client":"2","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188583192Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"task":"crutest"} {"msg":"The guest has requested activation with gen:66","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188618098Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464} {"msg":"417760d2-a386-483a-b03b-119b2bbbe63c active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188673791Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188703287Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"client":"0","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188721985Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"client":"1","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188739203Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"client":"2","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188755971Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"io task","client":"2","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188781397Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"io task","client":"0","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.18884264Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"io task","client":"1","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"ds_connection connected from Ok(127.0.0.1:65101)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188884864Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"io task","client":"2","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"ds_connection connected from Ok(127.0.0.1:58907)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188933868Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"io task","client":"1","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"ds_connection connected from Ok(127.0.0.1:64781)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.188969833Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"io task","client":"0","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.190154564Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"client":"1","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.19018924Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"client":"0","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.190734101Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.190835529Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.190865035Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"client":"2","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"[0]R flush_numbers[0..12]: [84, 116, 125, 125, 125, 115, 125, 122, 125, 125, 125, 125]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191420165Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"downstairs","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"[0]R generation[0..12]: [44, 60, 65, 65, 65, 58, 65, 62, 65, 65, 65, 65]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191451901Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"downstairs","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191472808Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"downstairs","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"[1]R flush_numbers[0..12]: [84, 116, 125, 125, 125, 115, 125, 122, 125, 125, 125, 125]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191492256Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"downstairs","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"[1]R generation[0..12]: [44, 60, 65, 65, 65, 58, 65, 62, 65, 65, 65, 65]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191513813Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"downstairs","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191534201Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"downstairs","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"[2]R flush_numbers[0..12]: [84, 116, 125, 125, 125, 115, 125, 122, 125, 125, 125, 125]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191553558Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"downstairs","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"[2]R generation[0..12]: [44, 60, 65, 65, 65, 58, 65, 62, 65, 65, 65, 65]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191572076Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"downstairs","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191591983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"downstairs","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"Max found gen is 66","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-11-06T20:13:55.191608551Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"downstairs","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"Generation requested: 66 >= found:66","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191628758Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"downstairs","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"Next flush: 126","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191645156Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"downstairs","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191661214Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"":"downstairs","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191677952Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.19169323Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"client":"0","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191710118Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"client":"1","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191726956Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"client":"2","":"downstairs","session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191744054Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191760122Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"417760d2-a386-483a-b03b-119b2bbbe63c is now active with session: 1a3ffe30-1d75-4759-902d-4793f35e4c44","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191784799Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191800067Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464,"session_id":"1a3ffe30-1d75-4759-902d-4793f35e4c44"} {"msg":"The guest has finished waiting for activation with:66","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191816125Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464} {"msg":"Activated sub_volume 417760d2-a386-483a-b03b-119b2bbbe63c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:55.191829853Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 120 len 24576 data: 11 6 5 6 5 6 02/30 Write block 80 len 20480 data: 7 6 6 6 5 03/30 Write block 147 len 40960 data: 2 2 2 2 3 3 3 4 5 4 04/30 Write block 438 len 4096 data: 6 05/30 Read block 171 len 40960 06/30 Read block 107 len 12288 07/30 Write block 212 len 36864 data: 3 4 4 4 5 5 5 6 7 08/30 Read block 578 len 8192 09/30 Read block 224 len 36864 10/30 Read block 412 len 24576 11/30 Flush 12/30 Write block 291 len 8192 data: 5 5 13/30 Read block 590 len 8192 14/30 Read block 515 len 40960 15/30 Write block 113 len 8192 data: 8 10 16/30 Write block 528 len 8192 data: 8 6 17/30 Write block 130 len 32768 data: 10 10 10 9 8 7 9 8 18/30 Write block 345 len 36864 data: 4 4 3 3 3 4 5 4 6 19/30 Read block 287 len 20480 20/30 Read block 262 len 20480 21/30 Read block 481 len 24576 22/30 Read block 491 len 8192 23/30 Read block 399 len 36864 24/30 Read block 124 len 28672 25/30 Read block 578 len 4096 26/30 Read block 213 len 12288 27/30 Read block 187 len 8192 28/30 Read block 576 len 4096 29/30 Read block 354 len 32768 30/30 Read block 240 len 32768 ---------------------------------------------------------------- Crucible gen:66 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 5 Done Done Done false 1002 Acked Write 10 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Read 10 Done Done Done false 1005 Acked Read 3 Done Done Sent false 1006 Acked Write 9 Done Done Sent false 1007 Acked Read 2 Done Done Sent false 1008 Acked Read 9 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 2 Done Done Sent false 1012 Acked Read 2 Done Done Sent false 1013 Acked Read 10 Done Done Sent false 1014 Acked Write 2 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Write 8 Done Done Sent false 1017 Acked Write 9 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Read 5 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Read 2 Done Done Sent false 1027 Acked Read 1 Done Done Sent false 1028 Acked Read 8 Done Done Sent false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 25 26 Done 30 29 5 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1010)) Some(JobId(1010)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:55.209927703Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1464} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 60 60 60 116 116 116 F F F 2 040-059 65 65 65 125 125 125 F F F 3 060-079 65 65 65 125 125 125 F F F 4 080-099 66 66 65 126 126 125 F F T <--- 5 100-119 58 58 58 115 115 115 T T F <--- 6 120-139 66 66 65 126 126 125 T T T <--- 7 140-159 66 66 62 126 126 122 F F T <--- 8 160-179 65 65 65 125 125 125 F F F 9 180-199 65 65 65 125 125 125 F F F 10 200-219 66 66 65 126 126 125 F F F <--- 11 220-239 66 66 65 126 126 125 F F F <--- 12 240-259 64 64 64 124 124 124 F F F 13 260-279 65 65 65 125 125 125 F F F 14 280-299 60 60 60 116 116 116 T T F <--- 15 300-319 65 65 65 125 125 125 F F F 16 320-339 61 61 61 120 120 120 F F F 17 340-359 62 62 62 122 122 122 T T F <--- 18 360-379 62 62 62 122 122 122 F F F 19 380-399 58 58 58 115 115 115 F F F 20 400-419 60 60 60 119 119 119 F F F 21 420-439 66 66 62 126 126 121 F F T <--- 22 440-459 65 65 65 125 125 125 F F F 23 460-479 62 62 62 121 121 121 F F F 24 480-499 63 63 63 123 123 123 F F F 25 500-519 62 62 62 121 121 121 F F F 26 520-539 57 57 57 114 114 114 T T F <--- 27 540-559 63 63 63 123 123 123 F F F 28 560-579 62 62 62 121 121 121 F F F 29 580-599 64 64 64 124 124 124 F F F Max gen: 66, Max flush: 126 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 33 Verifying data now No change after verify Loop: 33 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 60 60 60 116 116 116 F F F 2 040-059 65 65 65 125 125 125 F F F 3 060-079 65 65 65 125 125 125 F F F 4 080-099 66 66 66 126 126 126 F F F 5 100-119 67 67 67 127 127 127 F F F 6 120-139 67 67 67 127 127 127 F F F 7 140-159 66 66 66 126 126 126 F F F 8 160-179 65 65 65 125 125 125 F F F 9 180-199 65 65 65 125 125 125 F F F 10 200-219 66 66 66 126 126 126 F F F 11 220-239 66 66 66 126 126 126 F F F 12 240-259 64 64 64 124 124 124 F F F 13 260-279 65 65 65 125 125 125 F F F 14 280-299 67 67 67 127 127 127 F F F 15 300-319 65 65 65 125 125 125 F F F 16 320-339 61 61 61 120 120 120 F F F 17 340-359 67 67 67 127 127 127 F F F 18 360-379 62 62 62 122 122 122 F F F 19 380-399 58 58 58 115 115 115 F F F 20 400-419 60 60 60 119 119 119 F F F 21 420-439 66 66 66 126 126 126 F F F 22 440-459 65 65 65 125 125 125 F F F 23 460-479 62 62 62 121 121 121 F F F 24 480-499 63 63 63 123 123 123 F F F 25 500-519 62 62 62 121 121 121 F F F 26 520-539 67 67 67 127 127 127 F F F 27 540-559 63 63 63 123 123 123 F F F 28 560-579 62 62 62 121 121 121 F F F 29 580-599 64 64 64 124 124 124 F F F Max gen: 67, Max flush: 127 Begin loop 34 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.602250933Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:59.645701759Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.646100559Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.677379948Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.677954315Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.678014298Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.678032776Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.678055693Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472} {"msg":"Crucible 81cc6243-3737-4ae1-96af-41f1e154177c has session id: a2a8ff84-a6dc-4bf8-afb6-f13760336d4a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.678596385Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"Upstairs opts: Upstairs UUID: 81cc6243-3737-4ae1-96af-41f1e154177c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.678641149Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"Crucible stats registered with UUID: 81cc6243-3737-4ae1-96af-41f1e154177c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.678661746Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:59.678681874Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.678901316Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"io task","client":"1","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.678939861Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"io task","client":"2","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.678958609Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"io task","client":"0","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.679169092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"task":"crutest"} {"msg":"The guest has requested activation with gen:68","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.679193409Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472} {"msg":"81cc6243-3737-4ae1-96af-41f1e154177c active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.67926477Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.679289657Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"client":"0","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.679306945Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"client":"1","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.679325363Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"client":"2","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.679352749Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"io task","client":"2","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.679384025Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"io task","client":"0","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.679418091Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"io task","client":"1","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"ds_connection connected from Ok(127.0.0.1:38933)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.679519388Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"io task","client":"2","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"ds_connection connected from Ok(127.0.0.1:47093)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.679653161Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"io task","client":"0","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"ds_connection connected from Ok(127.0.0.1:62381)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.679710064Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"io task","client":"1","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.680825784Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"client":"1","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.680874658Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"client":"2","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.680948918Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"client":"0","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681463113Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681492929Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"[0]R flush_numbers[0..12]: [84, 116, 125, 125, 126, 127, 127, 126, 125, 125, 126, 126]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681512267Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"downstairs","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"[0]R generation[0..12]: [44, 60, 65, 65, 66, 67, 67, 66, 65, 65, 66, 66]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681531925Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"downstairs","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681552172Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"downstairs","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"[1]R flush_numbers[0..12]: [84, 116, 125, 125, 126, 127, 127, 126, 125, 125, 126, 126]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.68157145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"downstairs","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"[1]R generation[0..12]: [44, 60, 65, 65, 66, 67, 67, 66, 65, 65, 66, 66]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681590147Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"downstairs","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681610015Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"downstairs","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"[2]R flush_numbers[0..12]: [84, 116, 125, 125, 126, 127, 127, 126, 125, 125, 126, 126]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681629042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"downstairs","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"[2]R generation[0..12]: [44, 60, 65, 65, 66, 67, 67, 66, 65, 65, 66, 66]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.68164765Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"downstairs","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681667518Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"downstairs","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"Max found gen is 68","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681687655Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"downstairs","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"Generation requested: 68 >= found:68","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681704853Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472Wait for a query_work_queue command to finish before sending IO ,"":"downstairs","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"Next flush: 128","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681742428Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"downstairs","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681759186Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"":"downstairs","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681776164Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681791382Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"client":"0","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.68180821Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"client":"1","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681824948Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"client":"2","":"downstairs","session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681841885Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681857943Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"81cc6243-3737-4ae1-96af-41f1e154177c is now active with session: a2a8ff84-a6dc-4bf8-afb6-f13760336d4a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681875861Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2025-11-06T20:13:59.681891089Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472,"session_id":"a2a8ff84-a6dc-4bf8-afb6-f13760336d4a"} {"msg":"The guest has finished waiting for activation with:68","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681910077Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472} {"msg":"Activated sub_volume 81cc6243-3737-4ae1-96af-41f1e154177c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:13:59.681924025Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 180 len 36864 data: 10 12 12 12 7 7 7 5 6 02/30 Write block 471 len 32768 data: 5 6 6 5 6 5 4 6 03/30 Read block 207 len 20480 04/30 Read block 170 len 24576 05/30 Write block 52 len 36864 data: 4 5 5 4 3 3 4 3 4 06/30 Write block 418 len 20480 data: 6 6 6 7 7 07/30 Read block 453 len 28672 08/30 Read block 583 len 8192 09/30 Read block 282 len 36864 10/30 Read block 28 len 32768 11/30 Read block 20 len 32768 12/30 Read block 42 len 16384 13/30 Write block 274 len 8192 data: 5 4 14/30 Read block 594 len 8192 15/30 Read block 6 len 32768 16/30 Read block 192 len 36864 17/30 Read block 205 len 20480 18/30 Read block 79 len 28672 19/30 Read block 352 len 36864 20/30 Read block 215 len 36864 21/30 Write block 248 len 8192 data: 4 4 22/30 Read block 488 len 20480 23/30 Write block 578 len 24576 data: 3 4 3 3 4 3 24/30 Read block 213 len 16384 25/30 Read block 25 len 32768 26/30 Write block 196 len 24576 data: 4 6 6 5 5 7 27/30 Read block 406 len 12288 28/30 Read block 294 len 36864 29/30 Write block 141 len 40960 data: 8 9 7 8 6 3 3 3 3 3 30/30 Write block 289 len 8192 data: 6 6 ---------------------------------------------------------------- Crucible gen:68 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Sent Done Done false 1001 Acked Write 8 Sent Done Done false 1002 Acked Read 5 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Read 8 Sent Done Done false 1011 Acked Read 4 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Read 9 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Read 5 Sent Done Done false 1022 Acked Write 6 Sent Done Done false 1023 Acked Read 4 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Read 3 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:13:59.701507638Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1472} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 60 60 60 116 116 116 F F F 2 040-059 65 65 65 125 125 125 F T T <--- 3 060-079 65 65 65 125 125 125 F T T <--- 4 080-099 66 66 66 126 126 126 F F F 5 100-119 67 67 67 127 127 127 F F F 6 120-139 67 67 67 127 127 127 F F F 7 140-159 66 66 66 126 126 126 F T T <--- 8 160-179 65 65 65 125 125 125 F F F 9 180-199 65 65 65 125 125 125 F T T <--- 10 200-219 66 66 66 126 126 126 F T T <--- 11 220-239 66 66 66 126 126 126 F F F 12 240-259 64 64 64 124 124 124 F T T <--- 13 260-279 65 65 65 125 125 125 F T T <--- 14 280-299 67 67 67 127 127 127 F T T <--- 15 300-319 65 65 65 125 125 125 F F F 16 320-339 61 61 61 120 120 120 F F F 17 340-359 67 67 67 127 127 127 F F F 18 360-379 62 62 62 122 122 122 F F F 19 380-399 58 58 58 115 115 115 F F F 20 400-419 60 60 60 119 119 119 F T T <--- 21 420-439 66 66 66 126 126 126 F T T <--- 22 440-459 65 65 65 125 125 125 F F F 23 460-479 62 62 62 121 121 121 F T T <--- 24 480-499 63 63 63 123 123 123 F F F 25 500-519 62 62 62 121 121 121 F F F 26 520-539 67 67 67 127 127 127 F F F 27 540-559 63 63 63 123 123 123 F F F 28 560-579 62 62 62 121 121 121 F T T <--- 29 580-599 64 64 64 124 124 124 F T T <--- Max gen: 67, Max flush: 127 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 34 Verifying data now No change after verify Loop: 34 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 60 60 60 116 116 116 F F F 2 040-059 68 68 68 128 128 128 F F F 3 060-079 68 68 68 128 128 128 F F F 4 080-099 66 66 66 126 126 126 F F F 5 100-119 67 67 67 127 127 127 F F F 6 120-139 67 67 67 127 127 127 F F F 7 140-159 68 68 68 128 128 128 F F F 8 160-179 65 65 65 125 125 125 F F F 9 180-199 68 68 68 128 128 128 F F F 10 200-219 68 68 68 128 128 128 F F F 11 220-239 66 66 66 126 126 126 F F F 12 240-259 68 68 68 128 128 128 F F F 13 260-279 68 68 68 128 128 128 F F F 14 280-299 68 68 68 128 128 128 F F F 15 300-319 65 65 65 125 125 125 F F F 16 320-339 61 61 61 120 120 120 F F F 17 340-359 67 67 67 127 127 127 F F F 18 360-379 62 62 62 122 122 122 F F F 19 380-399 58 58 58 115 115 115 F F F 20 400-419 68 68 68 128 128 128 F F F 21 420-439 68 68 68 128 128 128 F F F 22 440-459 65 65 65 125 125 125 F F F 23 460-479 68 68 68 128 128 128 F F F 24 480-499 63 63 63 123 123 123 F F F 25 500-519 62 62 62 121 121 121 F F F 26 520-539 67 67 67 127 127 127 F F F 27 540-559 63 63 63 123 123 123 F F F 28 560-579 68 68 68 128 128 128 F F F 29 580-599 68 68 68 128 128 128 F F F Max gen: 68, Max flush: 128 Begin loop 35 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.143325042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.189589073Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.190280336Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.19033073Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.190360406Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.190397501Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480} {"msg":"Crucible 37dd5a88-bfab-4fad-bc86-5ac917818dde has session id: cf11f209-dd41-4c70-b775-5542f090b44b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.190995636Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"Upstairs opts: Upstairs UUID: 37dd5a88-bfab-4fad-bc86-5ac917818dde, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.191039631Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"Crucible stats registered with UUID: 37dd5a88-bfab-4fad-bc86-5ac917818dde","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.191059838Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:04.191076266Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.1912813Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"io task","client":"0","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.191326184Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"io task","client":"1","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.1913574Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"io task","client":"2","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.191604849Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"task":"crutest"} {"msg":"The guest has requested activation with gen:70","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.191633266Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480} {"msg":"37dd5a88-bfab-4fad-bc86-5ac917818dde active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.191675091Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.191693928Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"client":"0","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.191711476Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"client":"1","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.191733493Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"client":"2","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.19175913Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"io task","client":"2","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.191792446Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"io task","client":"0","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.191822862Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"io task","client":"1","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"ds_connection connected from Ok(127.0.0.1:44841)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.191867676Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"io task","client":"2","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"ds_connection connected from Ok(127.0.0.1:61174)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.191969783Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"io task","client":"0","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"ds_connection connected from Ok(127.0.0.1:40727)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.192043814Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"io task","client":"1","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.193000924Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"client":"1","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.193111179Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"client":"0","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.193365478Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"client":"2","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.193594369Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.193682248Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"[0]R flush_numbers[0..12]: [84, 116, 128, 128, 126, 127, 127, 128, 125, 128, 128, 126]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.193896091Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"downstairs","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"[0]R generation[0..12]: [44, 60, 68, 68, 66, 67, 67, 68, 65, 68, 68, 66]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.193922647Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"downstairs","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.193943275Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"downstairs","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"[1]R flush_numbers[0..12]: [84, 116, 128, 128, 126, 127, 127, 128, 125, 128, 128, 126]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.193965832Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"downstairs","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"[1]R generation[0..12]: [44, 60, 68, 68, 66, 67, 67, 68, 65, 68, 68, 66]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.19398468Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"downstairs","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194004427Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"downstairs","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"[2]R flush_numbers[0..12]: [84, 116, 128, 128, 126, 127, 127, 128, 125, 128, 128, 126]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194023475Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"downstairs","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"[2]R generation[0..12]: [44, 60, 68, 68, 66, 67, 67, 68, 65, 68, 68, 66]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194041772Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"downstairs","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.19406141Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"downstairs","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"Max found gen is 69","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194077488Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"downstairs","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"Generation requested: 70 >= found:69","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194094426Z","Wait for a query_work_queue command to finish before sending IO hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"downstairs","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"Next flush: 129","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194163227Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"downstairs","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194189464Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"":"downstairs","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194207181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194223839Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"client":"0","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194240957Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"client":"1","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194258255Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"client":"2","":"downstairs","session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194275563Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.19429549Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"37dd5a88-bfab-4fad-bc86-5ac917818dde is now active with session: cf11f209-dd41-4c70-b775-5542f090b44b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194314368Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194333396Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480,"session_id":"cf11f209-dd41-4c70-b775-5542f090b44b"} {"msg":"The guest has finished waiting for activation with:70","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194350083Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480} {"msg":"Activated sub_volume 37dd5a88-bfab-4fad-bc86-5ac917818dde","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:04.194364092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 501 len 20480 data: 7 7 8 5 6 02/30 Read block 41 len 16384 03/30 Read block 440 len 36864 04/30 Write block 200 len 24576 data: 6 8 6 7 8 10 05/30 Read block 395 len 36864 06/30 Read block 313 len 20480 07/30 Read block 211 len 24576 08/30 Write block 436 len 16384 data: 7 6 7 9 09/30 Read block 394 len 4096 10/30 Read block 382 len 24576 11/30 Read block 145 len 20480 12/30 Write block 492 len 16384 data: 3 3 2 2 13/30 Write block 350 len 20480 data: 5 6 5 7 5 14/30 Write block 357 len 20480 data: 6 5 5 4 5 15/30 Write block 457 len 4096 data: 10 16/30 Write block 129 len 24576 data: 13 11 11 11 10 9 17/30 Read block 286 len 20480 18/30 Flush 19/30 Write block 38 len 24576 data: 10 9 10 9 8 7 20/30 Read block 57 len 40960 21/30 Read block 237 len 28672 22/30 Read block 172 len 40960 23/30 Read block 417 len 32768 24/30 Write block 168 len 36864 data: 6 6 3 3 3 2 3 4 6 25/30 Write block 180 len 36864 data: 11 13 13 13 8 8 8 6 7 26/30 Read block 319 len 8192 27/30 Read block 373 len 40960 28/30 Read block 469 len 4096 29/30 Write block 64 len 24576 data: 6 7 6 6 7 6 30/30 Write block 542 len 16384 data: 10 8 8 8 ---------------------------------------------------------------- Crucible gen:70 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 4 Done Done Sent false 1002 Acked Read 9 Done Done Sent false 1003 Acked Write 6 Done Done Sent false 1004 Acked Read 9 Done Done Sent false 1005 Acked Read 5 Done Done Sent false 1006 Acked Read 6 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Read 1 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Read 5 Done Done Sent false 1011 Acked Write 4 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Write 5 Done Done Sent false 1014 Acked Write 1 Done Done Sent false 1015 Acked Write 6 Done Done Sent false 1016 Acked Read 5 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 6 Done Done Sent false 1019 Acked Read 10 Done Done Sent false 1020 Acked Read 7 Done Done Sent false 1021 Acked Read 10 Done Done Sent false 1022 Acked Read 8 Done Done Sent false 1023 Acked Write 9 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Read 2 Done Done Sent false 1026 Acked Read 10 Done Done Sent false 1027 Acked Read 1 Done Done Sent false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 29 33 Done 28 28 1 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) Some(JobId(1017)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:04.212966428Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1480} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 60 60 60 116 116 116 T T F <--- 2 040-059 68 68 68 128 128 128 T T F <--- 3 060-079 68 68 68 128 128 128 T T F <--- 4 080-099 66 66 66 126 126 126 F F F 5 100-119 67 67 67 127 127 127 F F F 6 120-139 70 70 67 129 129 127 F F F <--- 7 140-159 68 68 68 128 128 128 F F F 8 160-179 65 65 65 125 125 125 T T F <--- 9 180-199 68 68 68 128 128 128 T T F <--- 10 200-219 70 70 68 129 129 128 F F F <--- 11 220-239 66 66 66 126 126 126 F F F 12 240-259 68 68 68 128 128 128 F F F 13 260-279 68 68 68 128 128 128 F F F 14 280-299 68 68 68 128 128 128 F F F 15 300-319 65 65 65 125 125 125 F F F 16 320-339 61 61 61 120 120 120 F F F 17 340-359 70 70 67 129 129 127 F F F <--- 18 360-379 70 70 62 129 129 122 F F F <--- 19 380-399 58 58 58 115 115 115 F F F 20 400-419 68 68 68 128 128 128 F F F 21 420-439 70 70 68 129 129 128 F F F <--- 22 440-459 70 70 65 129 129 125 F F F <--- 23 460-479 68 68 68 128 128 128 F F F 24 480-499 70 70 63 129 129 123 F F F <--- 25 500-519 70 70 62 129 129 121 F F T <--- 26 520-539 67 67 67 127 127 127 F F F 27 540-559 63 63 63 123 123 123 T T F <--- 28 560-579 68 68 68 128 128 128 F F F 29 580-599 68 68 68 128 128 128 F F F Max gen: 70, Max flush: 129 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 35 Verifying data now No change after verify Loop: 35 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 F F F 1 020-039 71 71 71 130 130 130 F F F 2 040-059 71 71 71 130 130 130 F F F 3 060-079 71 71 71 130 130 130 F F F 4 080-099 66 66 66 126 126 126 F F F 5 100-119 67 67 67 127 127 127 F F F 6 120-139 70 70 70 129 129 129 F F F 7 140-159 68 68 68 128 128 128 F F F 8 160-179 71 71 71 130 130 130 F F F 9 180-199 71 71 71 130 130 130 F F F 10 200-219 70 70 70 129 129 129 F F F 11 220-239 66 66 66 126 126 126 F F F 12 240-259 68 68 68 128 128 128 F F F 13 260-279 68 68 68 128 128 128 F F F 14 280-299 68 68 68 128 128 128 F F F 15 300-319 65 65 65 125 125 125 F F F 16 320-339 61 61 61 120 120 120 F F F 17 340-359 70 70 70 129 129 129 F F F 18 360-379 70 70 70 129 129 129 F F F 19 380-399 58 58 58 115 115 115 F F F 20 400-419 68 68 68 128 128 128 F F F 21 420-439 70 70 70 129 129 129 F F F 22 440-459 70 70 70 129 129 129 F F F 23 460-479 68 68 68 128 128 128 F F F 24 480-499 70 70 70 129 129 129 F F F 25 500-519 70 70 70 129 129 129 F F F 26 520-539 67 67 67 127 127 127 F F F 27 540-559 71 71 71 130 130 130 F F F 28 560-579 68 68 68 128 128 128 F F F 29 580-599 68 68 68 128 128 128 F F F Max gen: 71, Max flush: 130 Begin loop 36 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.629106698Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.674301254Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.674908517Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.674957411Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.674973839Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.674996046Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488} {"msg":"Crucible 2de44a64-235a-4f56-9465-9a6b9d1b9431 has session id: 3f84992c-1e41-4a7d-9c69-c95ff528eca3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.675456778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"Upstairs opts: Upstairs UUID: 2de44a64-235a-4f56-9465-9a6b9d1b9431, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.675516801Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"Crucible stats registered with UUID: 2de44a64-235a-4f56-9465-9a6b9d1b9431","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.675547567Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:08.675573523Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.675813483Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.675848829Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"io task","client":"0","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.675867506Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"io task","client":"1","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.676029236Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"task":"crutest"} {"msg":"The guest has requested activation with gen:72","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.676052293Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488} {"msg":"2de44a64-235a-4f56-9465-9a6b9d1b9431 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.676120925Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.676140432Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"client":"0","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.67615747Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"client":"1","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.676178797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"client":"2","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.676195345Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.676212533Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"io task","client":"0","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.67623654Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"io task","client":"1","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"ds_connection connected from Ok(127.0.0.1:46192)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.676277605Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"io task","client":"2","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"ds_connection connected from Ok(127.0.0.1:42890)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.676323969Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"io task","client":"1","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"ds_connection connected from Ok(127.0.0.1:60978)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.676360104Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"io task","client":"0","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.677523008Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"client":"0","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.677624145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"client":"2","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.678179905Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.678211431Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.678525971Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"client":"1","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"[0]R flush_numbers[0..12]: [84, 130, 130, 130, 126, 127, 129, 128, 130, 130, 129, 126]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679054405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"downstairs","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"[0]R generation[0..12]: [44, 71, 71, 71, 66, 67, 70, 68, 71, 71, 70, 66]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679078442Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"downstairs","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679098909Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"downstairs","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"[1]R flush_numbers[0..12]: [84, 130, 130, 130, 126, 127, 129, 128, 130, 130, 129, 126]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679119177Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"downstairs","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"[1]R generation[0..12]: [44, 71, 71, 71, 66, 67, 70, 68, 71, 71, 70, 66]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679137865Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"downstairs","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679157602Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"downstairs","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"[2]R flush_numbers[0..12]: [84, 130, 130, 130, 126, 127, 129, 128, 130, 130, 129, 126]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679176629Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"downstairs","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"[2]R generation[0..12]: [44, 71, 71, 71, 66, 67, 70, 68, 71, 71, 70, 66]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679195237Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"downstairs","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679214925Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"downstairs","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"Max found gen is 72","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679231163Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"downstairs","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"Generation requested: 72 >= found:72","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-11-06T20:14:08.67925197Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"downstairs","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"Next flush: 131","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679272877Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"downstairs","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679289186Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"":"downstairs","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679306273Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679321541Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"client":"0","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679338569Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"client":"1","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679355547Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"client":"2","":"downstairs","session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679373775Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679390673Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"2de44a64-235a-4f56-9465-9a6b9d1b9431 is now active with session: 3f84992c-1e41-4a7d-9c69-c95ff528eca3","v":0,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 name":"crucible","level":30,"time":"2025-11-06T20:14:08.67941202Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679427538Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488,"session_id":"3f84992c-1e41-4a7d-9c69-c95ff528eca3"} {"msg":"The guest has finished waiting for activation with:72","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679443746Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488} {"msg":"Activated sub_volume 2de44a64-235a-4f56-9465-9a6b9d1b9431","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:08.679457894Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 348 len 8192 data: 4 4 02/30 Flush 03/30 Write block 273 len 12288 data: 9 6 5 04/30 Read block 531 len 28672 05/30 Write block 160 len 20480 data: 4 4 4 3 4 06/30 Write block 194 len 36864 data: 4 3 5 7 7 6 7 9 7 07/30 Write block 418 len 36864 data: 7 7 7 8 8 6 8 8 9 08/30 Write block 40 len 24576 data: 11 10 9 8 8 9 09/30 Write block 274 len 12288 data: 7 6 5 10/30 Write block 306 len 36864 data: 6 8 7 7 7 8 7 8 8 11/30 Write block 19 len 16384 data: 2 4 4 5 12/30 Write block 101 len 28672 data: 4 5 4 5 4 4 5 13/30 Read block 438 len 28672 14/30 Write block 391 len 32768 data: 6 7 6 5 5 5 6 5 15/30 Write block 250 len 36864 data: 6 6 6 6 7 6 6 5 6 16/30 Write block 127 len 12288 data: 8 11 14 17/30 Write block 449 len 32768 data: 8 9 11 9 9 11 12 12 18/30 Read block 504 len 20480 19/30 Write block 130 len 36864 data: 12 12 12 11 10 8 10 9 8 20/30 Write block 160 len 4096 data: 5 21/30 Read block 416 len 8192 22/30 Read block 329 len 40960 23/30 Write block 105 len 24576 data: 5 5 6 5 5 6 24/30 Read block 73 len 28672 25/30 Write block 239 len 36864 data: 7 7 8 8 8 8 7 6 4 26/30 Write block 205 len 28672 data: 11 8 7 5 4 4 3 27/30 Write block 586 len 20480 data: 7 8 11 13 12 28/30 Read block 125 len 12288 29/30 Read block 489 len 8192 30/30 Read block 202 len 40960 ---------------------------------------------------------------- Crucible gen:72 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 3 Done Sent Done false 1003 Acked Read 7 Done Sent Done false 1004 Acked Write 5 Done Sent Done false 1005 Acked Write 9 Done Sent Done false 1006 Acked Write 9 Done Sent Done false 1007 Acked Write 6 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Write 9 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Write 7 Done Sent Done false 1012 Acked Read 7 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 9 Done Sent Done false 1015 Acked Write 3 Done Sent Done false 1016 Acked Write 8 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Write 1 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Read 7 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Write 5 Done Sent Done false 1027 Acked Read 3 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Read 10 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 0 28 Done 28 0 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1001)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:08.697523948Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1488} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 44 44 44 84 84 84 T F T <--- 1 020-039 71 71 71 130 130 130 T F T <--- 2 040-059 71 71 71 130 130 130 T F T <--- 3 060-079 71 71 71 130 130 130 F F F 4 080-099 66 66 66 126 126 126 F F F 5 100-119 67 67 67 127 127 127 T F T <--- 6 120-139 70 70 70 129 129 129 T F T <--- 7 140-159 68 68 68 128 128 128 F F F 8 160-179 71 71 71 130 130 130 T F T <--- 9 180-199 71 71 71 130 130 130 T F T <--- 10 200-219 70 70 70 129 129 129 T F T <--- 11 220-239 66 66 66 126 126 126 T F T <--- 12 240-259 68 68 68 128 128 128 T F T <--- 13 260-279 68 68 68 128 128 128 T F T <--- 14 280-299 68 68 68 128 128 128 F F F 15 300-319 65 65 65 125 125 125 T F T <--- 16 320-339 61 61 61 120 120 120 F F F 17 340-359 72 72 72 131 131 131 F F F 18 360-379 70 70 70 129 129 129 F F F 19 380-399 58 58 58 115 115 115 T F T <--- 20 400-419 68 68 68 128 128 128 T F T <--- 21 420-439 70 70 70 129 129 129 T F T <--- 22 440-459 70 70 70 129 129 129 T F T <--- 23 460-479 68 68 68 128 128 128 F F F 24 480-499 70 70 70 129 129 129 F F F 25 500-519 70 70 70 129 129 129 F F F 26 520-539 67 67 67 127 127 127 F F F 27 540-559 71 71 71 130 130 130 F F F 28 560-579 68 68 68 128 128 128 F F F 29 580-599 68 68 68 128 128 128 T F T <--- Max gen: 72, Max flush: 131 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 36 Verifying data now No change after verify Loop: 36 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 132 132 132 F F F 1 020-039 73 73 73 132 132 132 F F F 2 040-059 73 73 73 132 132 132 F F F 3 060-079 71 71 71 130 130 130 F F F 4 080-099 66 66 66 126 126 126 F F F 5 100-119 73 73 73 132 132 132 F F F 6 120-139 73 73 73 132 132 132 F F F 7 140-159 68 68 68 128 128 128 F F F 8 160-179 73 73 73 132 132 132 F F F 9 180-199 73 73 73 132 132 132 F F F 10 200-219 73 73 73 132 132 132 F F F 11 220-239 73 73 73 132 132 132 F F F 12 240-259 73 73 73 132 132 132 F F F 13 260-279 73 73 73 132 132 132 F F F 14 280-299 68 68 68 128 128 128 F F F 15 300-319 73 73 73 132 132 132 F F F 16 320-339 61 61 61 120 120 120 F F F 17 340-359 72 72 72 131 131 131 F F F 18 360-379 70 70 70 129 129 129 F F F 19 380-399 73 73 73 132 132 132 F F F 20 400-419 73 73 73 132 132 132 F F F 21 420-439 73 73 73 132 132 132 F F F 22 440-459 73 73 73 132 132 132 F F F 23 460-479 68 68 68 128 128 128 F F F 24 480-499 70 70 70 129 129 129 F F F 25 500-519 70 70 70 129 129 129 F F F 26 520-539 67 67 67 127 127 127 F F F 27 540-559 71 71 71 130 130 130 F F F 28 560-579 68 68 68 128 128 128 F F F 29 580-599 73 73 73 132 132 132 F F F Max gen: 73, Max flush: 132 Begin loop 37 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.128530744Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.173774704Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.174389366Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.17444225Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.174464517Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.174486854Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496} {"msg":"Crucible 0aa4a56a-5874-4965-a130-6e040269ee00 has session id: 589d3751-7aca-4b42-9a8d-6f517150c6d5","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.174976332Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"Upstairs opts: Upstairs UUID: 0aa4a56a-5874-4965-a130-6e040269ee00, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175033805Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"Crucible stats registered with UUID: 0aa4a56a-5874-4965-a130-6e040269ee00","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175056532Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:13.17507351Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175326918Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"io task","client":"1","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175364634Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"io task","client":"0","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175396Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"io task","client":"2","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175571597Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"task":"crutest"} {"msg":"The guest has requested activation with gen:74","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175592525Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496} {"msg":"0aa4a56a-5874-4965-a130-6e040269ee00 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175691752Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.17571205Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"client":"0","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175732767Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"client":"1","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175750095Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"client":"2","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175766953Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"io task","client":"2","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175784151Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"io task","client":"0","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175801228Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"io task","client":"1","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"ds_connection connected from Ok(127.0.0.1:63182)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175917444Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"io task","client":"2","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"ds_connection connected from Ok(127.0.0.1:58169)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175962428Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"io task","client":"1","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"ds_connection connected from Ok(127.0.0.1:50889)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.175996924Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"io task","client":"0","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.177268154Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"client":"1","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.177316358Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"client":"0","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.177834302Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.17793116Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.177954777Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"client":"2","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"[0]R flush_numbers[0..12]: [132, 132, 132, 130, 126, 132, 132, 128, 132, 132, 132, 132]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.17848922Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"downstairs","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"[0]R generation[0..12]: [73, 73, 73, 71, 66, 73, 73, 68, 73, 73, 73, 73]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178521686Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"downstairs","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178545743Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"downstairs","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"[1]R flush_numbers[0..12]: [132, 132, 132, 130, 126, 132, 132, 128, 132, 132, 132, 132]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.17856532Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"downstairs","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"[1]R generation[0..12]: [73, 73, 73, 71, 66, 73, 73, 68, 73, 73, 73, 73]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178583608Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"downstairs","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178603526Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"downstairs","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"[2]R flush_numbers[0..12]: [132, 132, 132, 130, 126, 132, 132, 128, 132, 132, 132, 132]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178622623Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"downstairs","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"[2]R generation[0..12]: [73, 73, 73, 71, 66, 73, 73, 68, 73, 73, 73, 73]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178640761Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"downstairs","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178660538Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"downstairs","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"Max found gen is 74","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178676806Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"downstairs","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"Generation requested: 74 >= found:74","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178693524Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"downstairs","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"Next flush: 133","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-11-06T20:14:13.178709612Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"downstairs","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178733019Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"":"downstairs","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178750147Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178765515Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"client":"0","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178782443Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"client":"1","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178799271Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"client":"2","":"downstairs","session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178820058Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178836386Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"0aa4a56a-5874-4965-a130-6e040269ee00 is now active with session: 589d3751-7aca-4b42-9a8d-6f517150c6d5","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178854204Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178869402Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496,"session_id":"589d3751-7aca-4b42-9a8d-6f517150c6d5"} {"msg":"The guest has finished waiting for activation with:74","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.17888514Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496} {"msg":"Activated sub_volume 0aa4a56a-5874-4965-a130-6e040269ee00","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:13.178898958Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 76 len 28672 data: 5 5 7 7 8 7 7 02/30 Read block 400 len 12288 03/30 Write block 197 len 20480 data: 8 8 7 8 10 04/30 Flush 05/30 Write block 233 len 4096 data: 6 06/30 Write block 31 len 20480 data: 5 5 6 8 8 07/30 Read block 164 len 32768 08/30 Read block 546 len 20480 09/30 Read block 442 len 40960 10/30 Write block 399 len 28672 data: 6 6 8 7 6 6 6 11/30 Read block 210 len 12288 12/30 Write block 549 len 12288 data: 7 8 7 13/30 Read block 348 len 28672 14/30 Write block 35 len 32768 data: 9 10 9 11 10 12 11 10 15/30 Read block 503 len 36864 16/30 Read block 449 len 36864 17/30 Write block 138 len 12288 data: 9 7 8 18/30 Write block 223 len 24576 data: 7 4 3 3 3 4 19/30 Read block 514 len 16384 20/30 Write block 64 len 12288 data: 7 8 7 21/30 Read block 449 len 8192 22/30 Read block 133 len 8192 23/30 Write block 487 len 28672 data: 4 5 4 3 2 4 4 24/30 Write block 297 len 28672 data: 4 4 5 5 5 6 6 25/30 Write block 504 len 20480 data: 6 7 6 4 6 26/30 Write block 118 len 28672 data: 11 11 12 7 6 7 6 27/30 Read block 68 len 32768 28/30 Write block 450 len 8192 data: 10 12 29/30 Write block 179 len 8192 data: 10 12 30/30 Read block 31 len 32768 ---------------------------------------------------------------- Crucible gen:74 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 3 Done Done Sent false 1002 Acked Write 5 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 1 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Read 8 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Read 10 Done Done Sent false 1009 Acked Write 7 Done Done Sent false 1010 Acked Read 3 Done Done Sent false 1011 Acked Write 3 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Write 8 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Write 3 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Write 7 Done Done Sent false 1023 Acked Write 7 Done Done Sent false 1024 Acked Write 5 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Read 8 Done Done Sent false 1027 Acked Write 2 Done Done Sent false 1028 Acked Write 2 Done Done Sent false 1029 Acked Read 8 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 29 30 Done 29 30 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:13.197535Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1496} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 132 132 132 F F F 1 020-039 73 73 73 132 132 132 T T F <--- 2 040-059 73 73 73 132 132 132 T T F <--- 3 060-079 74 74 71 133 133 130 T T T <--- 4 080-099 74 74 66 133 133 126 F F T <--- 5 100-119 73 73 73 132 132 132 T T F <--- 6 120-139 73 73 73 132 132 132 T T F <--- 7 140-159 68 68 68 128 128 128 T T F <--- 8 160-179 73 73 73 132 132 132 T T F <--- 9 180-199 74 74 73 133 133 132 T T F <--- 10 200-219 74 74 73 133 133 132 F F F <--- 11 220-239 73 73 73 132 132 132 T T F <--- 12 240-259 73 73 73 132 132 132 F F F 13 260-279 73 73 73 132 132 132 F F F 14 280-299 68 68 68 128 128 128 T T F <--- 15 300-319 73 73 73 132 132 132 T T F <--- 16 320-339 61 61 61 120 120 120 F F F 17 340-359 72 72 72 131 131 131 F F F 18 360-379 70 70 70 129 129 129 F F F 19 380-399 73 73 73 132 132 132 T T F <--- 20 400-419 73 73 73 132 132 132 T T F <--- 21 420-439 73 73 73 132 132 132 F F F 22 440-459 73 73 73 132 132 132 T T F <--- 23 460-479 68 68 68 128 128 128 F F F 24 480-499 70 70 70 129 129 129 T T F <--- 25 500-519 70 70 70 129 129 129 T T F <--- 26 520-539 67 67 67 127 127 127 F F F 27 540-559 71 71 71 130 130 130 T T F <--- 28 560-579 68 68 68 128 128 128 F F F 29 580-599 73 73 73 132 132 132 F F F Max gen: 74, Max flush: 133 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 37 Verifying data now No change after verify Loop: 37 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 132 132 132 F F F 1 020-039 75 75 75 134 134 134 F F F 2 040-059 75 75 75 134 134 134 F F F 3 060-079 75 75 75 134 134 134 F F F 4 080-099 74 74 74 133 133 133 F F F 5 100-119 75 75 75 134 134 134 F F F 6 120-139 75 75 75 134 134 134 F F F 7 140-159 75 75 75 134 134 134 F F F 8 160-179 75 75 75 134 134 134 F F F 9 180-199 75 75 75 134 134 134 F F F 10 200-219 74 74 74 133 133 133 F F F 11 220-239 75 75 75 134 134 134 F F F 12 240-259 73 73 73 132 132 132 F F F 13 260-279 73 73 73 132 132 132 F F F 14 280-299 75 75 75 134 134 134 F F F 15 300-319 75 75 75 134 134 134 F F F 16 320-339 61 61 61 120 120 120 F F F 17 340-359 72 72 72 131 131 131 F F F 18 360-379 70 70 70 129 129 129 F F F 19 380-399 75 75 75 134 134 134 F F F 20 400-419 75 75 75 134 134 134 F F F 21 420-439 73 73 73 132 132 132 F F F 22 440-459 75 75 75 134 134 134 F F F 23 460-479 68 68 68 128 128 128 F F F 24 480-499 75 75 75 134 134 134 F F F 25 500-519 75 75 75 134 134 134 F F F 26 520-539 67 67 67 127 127 127 F F F 27 540-559 75 75 75 134 134 134 F F F 28 560-579 68 68 68 128 128 128 F F F 29 580-599 73 73 73 132 132 132 F F F Max gen: 75, Max flush: 134 Begin loop 38 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.654651247Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:17.698456559Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.69883992Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.730323154Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.730919369Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.730966753Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.730982871Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.731004708Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504} {"msg":"Crucible ee0af8ac-00aa-40c4-9b05-b132f249be08 has session id: 5853b737-4aed-4e0b-ad15-e60700768306","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.731677103Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"Upstairs opts: Upstairs UUID: ee0af8ac-00aa-40c4-9b05-b132f249be08, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.731748044Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"Crucible stats registered with UUID: ee0af8ac-00aa-40c4-9b05-b132f249be08","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.73177781Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:17.731798698Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732011541Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"io task","client":"0","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732059675Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"io task","client":"1","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732084692Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"io task","client":"2","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732324992Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"task":"crutest"} {"msg":"The guest has requested activation with gen:76","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732354728Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504} {"msg":"ee0af8ac-00aa-40c4-9b05-b132f249be08 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732425639Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732445366Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"client":"0","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732462474Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"client":"1","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732479182Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"client":"2","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.73249584Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"io task","client":"2","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732522307Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"io task","client":"0","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732552303Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"io task","client":"1","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"ds_connection connected from Ok(127.0.0.1:51470)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732590928Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"io task","client":"2","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"ds_connection connected from Ok(127.0.0.1:43907)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732624684Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"io task","client":"1","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"ds_connection connected from Ok(127.0.0.1:35054)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.732659259Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"io task","client":"0","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.733935119Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"client":"1","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734046405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"client":"0","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734099938Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"client":"2","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734494198Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734610374Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"[0]R flush_numbers[0..12]: [132, 134, 134, 134, 133, 134, 134, 134, 134, 134, 133, 134]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734635661Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"downstairs","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"[0]R generation[0..12]: [73, 75, 75, 75, 74, 75, 75, 75, 75, 75, 74, 75]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734655908Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"downstairs","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734676265Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"downstairs","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"[1]R flush_numbers[0..12]: [132, 134, 134, 134, 133, 134, 134, 134, 134, 134, 133, 134]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734695563Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"downstairs","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"[1]R generation[0..12]: [73, 75, 75, 75, 74, 75, 75, 75, 75, 75, 74, 75]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734713891Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"downstairs","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734733808Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"downstairs","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"[2]R flush_numbers[0..12]: [132, 134, 134, 134, 133, 134, 134, 134, 134, 134, 133, 134]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734757085Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"downstairs","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"[2]R generation[0..12]: [73, 75, 75, 75, 74, 75, 75, 75, 75, 75, 74, 75]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734776873Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"downstairs","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.73479662Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"downstairs","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"Max found gen is 76","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734813638Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"downstairs","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"Generation requested: 76 >= found:76","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734830736Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"downstairs","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"Next flush: 135","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734846804Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"downstairs","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {Wait for a query_work_queue command to finish before sending IO "msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734868631Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"":"downstairs","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734885469Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734900477Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"client":"0","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734917015Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"client":"1","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734933513Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"client":"2","":"downstairs","session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734950531Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734966559Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"ee0af8ac-00aa-40c4-9b05-b132f249be08 is now active with session: 5853b737-4aed-4e0b-ad15-e60700768306","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734984317Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.734999585Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504,"session_id":"5853b737-4aed-4e0b-ad15-e60700768306"} {"msg":"The guest has finished waiting for activation with:76","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.735015543Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504} {"msg":"Activated sub_volume ee0af8ac-00aa-40c4-9b05-b132f249be08","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:17.73503544Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 41 len 12288 data: 12 11 9 02/30 Read block 357 len 36864 03/30 Read block 312 len 36864 04/30 Write block 538 len 20480 data: 7 7 7 8 11 05/30 Flush 06/30 Write block 335 len 28672 data: 6 6 6 7 7 5 5 07/30 Read block 178 len 28672 08/30 Write block 472 len 24576 data: 7 7 6 7 6 5 09/30 Write block 393 len 8192 data: 7 6 10/30 Write block 514 len 20480 data: 6 6 6 9 7 11/30 Write block 333 len 12288 data: 8 7 7 12/30 Read block 348 len 32768 13/30 Write block 576 len 32768 data: 4 5 4 5 4 4 5 4 14/30 Write block 98 len 32768 data: 5 6 5 5 6 5 6 6 15/30 Read block 286 len 28672 16/30 Write block 174 len 40960 data: 4 5 7 7 7 11 13 14 14 14 17/30 Read block 356 len 16384 18/30 Write block 197 len 16384 data: 9 9 8 9 19/30 Read block 319 len 4096 20/30 Write block 199 len 28672 data: 9 10 11 8 8 9 12 21/30 Read block 432 len 20480 22/30 Write block 415 len 16384 data: 5 6 6 8 23/30 Read block 444 len 20480 24/30 Write block 418 len 32768 data: 9 8 8 9 9 7 9 9 25/30 Write block 334 len 28672 data: 8 8 7 7 8 8 6 26/30 Write block 157 len 40960 data: 4 4 6 6 5 5 4 5 5 5 27/30 Read block 583 len 4096 28/30 Read block 321 len 32768 29/30 Read block 557 len 40960 30/30 Write block 357 len 12288 data: 7 6 6 ---------------------------------------------------------------- Crucible gen:76 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Sent Done Done false 1001 Acked Read 9 Sent Done Done false 1002 Acked Read 9 Sent Done Done false 1003 Acked Write 5 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 7 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Write 5 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Write 8 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Read 7 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Write 4 Sent Done Done false 1018 Acked Read 1 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Read 5 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Read 5 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Write 7 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1004)) Some(JobId(1004)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:17.75424451Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1504} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 132 132 132 F F F 1 020-039 75 75 75 134 134 134 F F F 2 040-059 75 76 76 134 135 135 F F F <--- 3 060-079 75 75 75 134 134 134 F F F 4 080-099 74 74 74 133 133 133 F T T <--- 5 100-119 75 75 75 134 134 134 F T T <--- 6 120-139 75 75 75 134 134 134 F F F 7 140-159 75 75 75 134 134 134 F T T <--- 8 160-179 75 75 75 134 134 134 F T T <--- 9 180-199 75 75 75 134 134 134 F T T <--- 10 200-219 74 74 74 133 133 133 F T T <--- 11 220-239 75 75 75 134 134 134 F F F 12 240-259 73 73 73 132 132 132 F F F 13 260-279 73 73 73 132 132 132 F F F 14 280-299 75 75 75 134 134 134 F F F 15 300-319 75 75 75 134 134 134 F F F 16 320-339 61 61 61 120 120 120 F T T <--- 17 340-359 72 72 72 131 131 131 F T T <--- 18 360-379 70 70 70 129 129 129 F F F 19 380-399 75 75 75 134 134 134 F T T <--- 20 400-419 75 75 75 134 134 134 F T T <--- 21 420-439 73 73 73 132 132 132 F T T <--- 22 440-459 75 75 75 134 134 134 F F F 23 460-479 68 68 68 128 128 128 F T T <--- 24 480-499 75 75 75 134 134 134 F F F 25 500-519 75 75 75 134 134 134 F T T <--- 26 520-539 67 76 76 127 135 135 F F F <--- 27 540-559 75 76 76 134 135 135 F F F <--- 28 560-579 68 68 68 128 128 128 F T T <--- 29 580-599 73 73 73 132 132 132 F T T <--- Max gen: 76, Max flush: 135 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 38 Verifying data now No change after verify Loop: 38 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 132 132 132 F F F 1 020-039 75 75 75 134 134 134 F F F 2 040-059 76 76 76 135 135 135 F F F 3 060-079 75 75 75 134 134 134 F F F 4 080-099 77 77 77 136 136 136 F F F 5 100-119 77 77 77 136 136 136 F F F 6 120-139 75 75 75 134 134 134 F F F 7 140-159 77 77 77 136 136 136 F F F 8 160-179 77 77 77 136 136 136 F F F 9 180-199 77 77 77 136 136 136 F F F 10 200-219 77 77 77 136 136 136 F F F 11 220-239 75 75 75 134 134 134 F F F 12 240-259 73 73 73 132 132 132 F F F 13 260-279 73 73 73 132 132 132 F F F 14 280-299 75 75 75 134 134 134 F F F 15 300-319 75 75 75 134 134 134 F F F 16 320-339 77 77 77 136 136 136 F F F 17 340-359 77 77 77 136 136 136 F F F 18 360-379 70 70 70 129 129 129 F F F 19 380-399 77 77 77 136 136 136 F F F 20 400-419 77 77 77 136 136 136 F F F 21 420-439 77 77 77 136 136 136 F F F 22 440-459 75 75 75 134 134 134 F F F 23 460-479 77 77 77 136 136 136 F F F 24 480-499 75 75 75 134 134 134 F F F 25 500-519 77 77 77 136 136 136 F F F 26 520-539 76 76 76 135 135 135 F F F 27 540-559 76 76 76 135 135 135 F F F 28 560-579 77 77 77 136 136 136 F F F 29 580-599 77 77 77 136 136 136 F F F Max gen: 77, Max flush: 136 Begin loop 39 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.224347739Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.269682408Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.270374091Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.270424234Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.270440982Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.270463489Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512} {"msg":"Crucible 3228985c-16d5-4930-971d-22c4164b9056 has session id: 24260d88-b360-4b85-bef0-0f553b09c861","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.27109138Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"Upstairs opts: Upstairs UUID: 3228985c-16d5-4930-971d-22c4164b9056, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.271150443Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"Crucible stats registered with UUID: 3228985c-16d5-4930-971d-22c4164b9056","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.271180499Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:22.271207056Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.271429858Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.271465203Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.27148727Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.271657019Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"task":"crutest"} {"msg":"The guest has requested activation with gen:78","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.27172361Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512} {"msg":"3228985c-16d5-4930-971d-22c4164b9056 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.271789962Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.27181082Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"client":"0","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.271828327Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"client":"1","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.271845125Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"client":"2","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.271861933Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.271879101Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.271898339Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"ds_connection connected from Ok(127.0.0.1:46912)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.272015094Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"io task","client":"0","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"ds_connection connected from Ok(127.0.0.1:40805)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.272252154Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"io task","client":"1","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"ds_connection connected from Ok(127.0.0.1:37318)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.272292779Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"io task","client":"2","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.273320619Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"client":"1","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.273426606Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"client":"0","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.273928623Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274036339Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274061746Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"client":"2","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"[0]R flush_numbers[0..12]: [132, 134, 135, 134, 136, 136, 134, 136, 136, 136, 136, 134]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274608157Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"downstairs","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"[0]R generation[0..12]: [73, 75, 76, 75, 77, 77, 75, 77, 77, 77, 77, 75]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274632514Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"downstairs","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274652681Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"downstairs","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"[1]R flush_numbers[0..12]: [132, 134, 135, 134, 136, 136, 134, 136, 136, 136, 136, 134]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274671749Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"downstairs","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"[1]R generation[0..12]: [73, 75, 76, 75, 77, 77, 75, 77, 77, 77, 77, 75]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274690177Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"downstairs","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274709664Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"downstairs","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"[2]R flush_numbers[0..12]: [132, 134, 135, 134, 136, 136, 134, 136, 136, 136, 136, 134]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274728822Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"downstairs","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"[2]R generation[0..12]: [73, 75, 76, 75, 77, 77, 75, 77, 77, 77, 77, 75]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.2747473Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"downstairs","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274767127Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"downstairs","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"Max found gen is 78","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274783395Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"downstairs","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"Generation requested: 78 >= found:78","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-11-06T20:14:22.274800543Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"downstairs","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"Next flush: 137","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.27482268Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"downstairs","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274843668Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"":"downstairs","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274861485Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274877053Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"client":"0","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274893941Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"client":"1","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274910749Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"client":"2","":"downstairs","session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274928047Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:14:22.274944365Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"3228985c-16d5-4930-971d-22c4164b9056 is now active with session: 24260d88-b360-4b85-bef0-0f553b09c861","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274966612Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.2749821Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512,"session_id":"24260d88-b360-4b85-bef0-0f553b09c861"} {"msg":"The guest has finished waiting for activation with:78","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.274998138Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512} {"msg":"Activated sub_volume 3228985c-16d5-4930-971d-22c4164b9056","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:22.275012246Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 178 len 16384 data: 8 12 14 15 02/30 Write block 41 len 36864 data: 13 12 10 9 10 9 10 8 7 03/30 Write block 174 len 4096 data: 5 04/30 Read block 370 len 32768 05/30 Read block 210 len 32768 06/30 Read block 6 len 24576 07/30 Write block 439 len 4096 data: 10 08/30 Write block 530 len 12288 data: 5 6 6 09/30 Write block 26 len 20480 data: 7 6 6 5 5 10/30 Write block 29 len 40960 data: 6 6 6 6 7 9 10 11 10 12 11/30 Read block 269 len 16384 12/30 Write block 387 len 24576 data: 5 6 6 6 7 8 13/30 Write block 371 len 24576 data: 2 2 2 2 2 2 14/30 Flush 15/30 Write block 22 len 12288 data: 6 6 6 16/30 Read block 168 len 28672 17/30 Read block 382 len 24576 18/30 Read block 423 len 16384 19/30 Read block 61 len 32768 20/30 Write block 102 len 24576 data: 7 6 7 7 6 7 21/30 Read block 114 len 4096 22/30 Flush 23/30 Write block 216 len 40960 data: 6 6 6 7 8 6 6 8 5 4 24/30 Write block 48 len 20480 data: 9 8 8 7 5 25/30 Read block 245 len 36864 26/30 Read block 297 len 16384 27/30 Write block 22 len 28672 data: 7 7 7 8 8 7 7 28/30 Read block 480 len 24576 29/30 Read block 426 len 36864 30/30 Write block 377 len 24576 data: 3 3 3 4 5 4 ---------------------------------------------------------------- Crucible gen:78 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 9 Done Done Sent false 1002 Acked Write 1 Done Done Sent false 1003 Acked Read 8 Done Done Sent false 1004 Acked Read 8 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Write 3 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Write 10 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Read 7 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Read 4 Done Done Sent false 1018 Acked Read 8 Done Done Sent false 1019 Acked Write 6 Done Done Sent false 1020 Acked Read 1 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 10 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Read 9 Done Done Sent false 1025 Acked Read 4 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Read 6 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1021)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:22.293963999Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1512} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 132 132 132 F F F 1 020-039 78 78 75 138 138 134 T T F <--- 2 040-059 78 78 76 137 137 135 T T F <--- 3 060-079 75 75 75 134 134 134 F F F 4 080-099 77 77 77 136 136 136 F F F 5 100-119 78 78 77 138 138 136 F F F <--- 6 120-139 75 75 75 134 134 134 F F F 7 140-159 77 77 77 136 136 136 F F F 8 160-179 78 78 77 137 137 136 F F T <--- 9 180-199 78 78 77 137 137 136 F F T <--- 10 200-219 77 77 77 136 136 136 T T F <--- 11 220-239 75 75 75 134 134 134 T T F <--- 12 240-259 73 73 73 132 132 132 F F F 13 260-279 73 73 73 132 132 132 F F F 14 280-299 75 75 75 134 134 134 F F F 15 300-319 75 75 75 134 134 134 F F F 16 320-339 77 77 77 136 136 136 F F F 17 340-359 77 77 77 136 136 136 F F F 18 360-379 78 78 70 137 137 129 T T F <--- 19 380-399 78 78 77 137 137 136 T T F <--- 20 400-419 77 77 77 136 136 136 F F F 21 420-439 78 78 77 137 137 136 F F F <--- 22 440-459 75 75 75 134 134 134 F F F 23 460-479 77 77 77 136 136 136 F F F 24 480-499 75 75 75 134 134 134 F F F 25 500-519 77 77 77 136 136 136 F F F 26 520-539 78 78 76 137 137 135 F F F <--- 27 540-559 76 76 76 135 135 135 F F F 28 560-579 77 77 77 136 136 136 F F F 29 580-599 77 77 77 136 136 136 F F F Max gen: 78, Max flush: 138 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 39 Verifying data now No change after verify Loop: 39 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 73 73 73 132 132 132 F F F 1 020-039 79 79 79 139 139 139 F F F 2 040-059 79 79 79 139 139 139 F F F 3 060-079 75 75 75 134 134 134 F F F 4 080-099 77 77 77 136 136 136 F F F 5 100-119 78 78 78 138 138 138 F F F 6 120-139 75 75 75 134 134 134 F F F 7 140-159 77 77 77 136 136 136 F F F 8 160-179 78 78 78 137 137 137 F F F 9 180-199 78 78 78 137 137 137 F F F 10 200-219 79 79 79 139 139 139 F F F 11 220-239 79 79 79 139 139 139 F F F 12 240-259 73 73 73 132 132 132 F F F 13 260-279 73 73 73 132 132 132 F F F 14 280-299 75 75 75 134 134 134 F F F 15 300-319 75 75 75 134 134 134 F F F 16 320-339 77 77 77 136 136 136 F F F 17 340-359 77 77 77 136 136 136 F F F 18 360-379 79 79 79 139 139 139 F F F 19 380-399 79 79 79 139 139 139 F F F 20 400-419 77 77 77 136 136 136 F F F 21 420-439 78 78 78 137 137 137 F F F 22 440-459 75 75 75 134 134 134 F F F 23 460-479 77 77 77 136 136 136 F F F 24 480-499 75 75 75 134 134 134 F F F 25 500-519 77 77 77 136 136 136 F F F 26 520-539 78 78 78 137 137 137 F F F 27 540-559 76 76 76 135 135 135 F F F 28 560-579 77 77 77 136 136 136 F F F 29 580-599 77 77 77 136 136 136 F F F Max gen: 79, Max flush: 139 Begin loop 40 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.693453106Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.738562993Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.739208981Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.739256665Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.739273713Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.73929924Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520} {"msg":"Crucible 278d7c5b-7240-44b7-ad1f-88458d65a49a has session id: fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.739831533Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"Upstairs opts: Upstairs UUID: 278d7c5b-7240-44b7-ad1f-88458d65a49a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.739889015Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"Crucible stats registered with UUID: 278d7c5b-7240-44b7-ad1f-88458d65a49a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.739918752Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:26.739944388Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.740103388Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"io task","client":"2","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.740154502Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"io task","client":"0","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.740185498Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"io task","client":"1","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.740290625Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"task":"crutest"} {"msg":"The guest has requested activation with gen:80","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.740320301Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520} {"msg":"278d7c5b-7240-44b7-ad1f-88458d65a49a active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.740383713Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.740422408Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"client":"0","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.740456314Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"client":"1","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.74048996Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"client":"2","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.740524095Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"io task","client":"2","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.7405652Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"io task","client":"0","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.740598406Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"io task","client":"1","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"ds_connection connected from Ok(127.0.0.1:61875)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.740632492Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"io task","client":"2","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"ds_connection connected from Ok(127.0.0.1:48723)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.740767875Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"io task","client":"0","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"ds_connection connected from Ok(127.0.0.1:33361)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.740808279Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"io task","client":"1","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742022526Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"client":"0","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742057592Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"client":"2","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742078279Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"client":"1","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742596534Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742701571Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"[0]R flush_numbers[0..12]: [132, 139, 139, 134, 136, 138, 134, 136, 137, 137, 139, 139]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742798389Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"downstairs","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"[0]R generation[0..12]: [73, 79, 79, 75, 77, 78, 75, 77, 78, 78, 79, 79]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742822036Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"downstairs","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742842153Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"downstairs","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"[1]R flush_numbers[0..12]: [132, 139, 139, 134, 136, 138, 134, 136, 137, 137, 139, 139]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742861011Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"downstairs","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"[1]R generation[0..12]: [73, 79, 79, 75, 77, 78, 75, 77, 78, 78, 79, 79]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742882738Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"downstairs","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742902546Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"downstairs","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"[2]R flush_numbers[0..12]: [132, 139, 139, 134, 136, 138, 134, 136, 137, 137, 139, 139]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742921503Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"downstairs","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"[2]R generation[0..12]: [73, 79, 79, 75, 77, 78, 75, 77, 78, 78, 79, 79]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742939751Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"downstairs","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742959059Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"downstairs","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"Max found gen is 80","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-11-06T20:14:26.742975457Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"downstairs","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"Generation requested: 80 >= found:80","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.742998184Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"downstairs","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"Next flush: 140","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.743014042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"downstairs","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.74302989Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"":"downstairs","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.743047008Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.743062395Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"client":"0","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.743079024Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"client":"1","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.743095671Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"client":"2","":"downstairs","session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.743112869Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2025-11-06T20:14:26.743128887Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"278d7c5b-7240-44b7-ad1f-88458d65a49a is now active with session: fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.743151614Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.743167022Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520,"session_id":"fe1026cc-b15e-4ddd-bc19-b21c5d4e3ae3"} {"msg":"The guest has finished waiting for activation with:80","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.74318285Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520} {"msg":"Activated sub_volume 278d7c5b-7240-44b7-ad1f-88458d65a49a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:26.743196669Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 236 len 8192 data: 6 6 02/30 Write block 45 len 32768 data: 11 10 11 10 9 9 8 6 03/30 Write block 229 len 16384 data: 5 5 6 6 04/30 Write block 531 len 24576 data: 7 7 8 8 9 9 05/30 Write block 491 len 28672 data: 3 5 5 3 3 2 2 06/30 Read block 294 len 24576 07/30 Write block 6 len 36864 data: 3 3 4 3 3 4 4 3 3 08/30 Write block 214 len 4096 data: 5 09/30 Write block 419 len 32768 data: 9 9 10 10 8 10 10 10 10/30 Flush 11/30 Write block 374 len 28672 data: 3 3 3 4 4 4 5 12/30 Write block 33 len 36864 data: 8 10 11 12 11 13 11 13 14 13/30 Read block 283 len 40960 14/30 Flush 15/30 Write block 276 len 28672 data: 6 7 7 7 6 6 6 16/30 Flush 17/30 Write block 557 len 36864 data: 6 5 4 3 4 4 4 4 4 18/30 Read block 32 len 28672 19/30 Read block 142 len 40960 20/30 Write block 136 len 36864 data: 11 10 10 8 9 9 10 8 9 21/30 Read block 362 len 4096 22/30 Read block 530 len 20480 23/30 Read block 319 len 24576 24/30 Flush 25/30 Write block 152 len 16384 data: 4 4 5 6 26/30 Read block 392 len 32768 27/30 Read block 159 len 24576 28/30 Write block 58 len 12288 data: 5 4 5 29/30 Write block 510 len 28672 data: 7 7 7 7 7 7 7 30/30 Read block 422 len 20480 ---------------------------------------------------------------- Crucible gen:80 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Write 4 Done Sent Done false 1003 Acked Write 6 Done Sent Done false 1004 Acked Write 7 Done Sent Done false 1005 Acked Read 6 Done Sent Done false 1006 Acked Write 9 Done Sent Done false 1007 Acked Write 1 Done Sent Done false 1008 Acked Write 8 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 7 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Read 10 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Read 10 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Read 1 Done Sent Done false 1021 Acked Read 5 Done Sent Done false 1022 Acked Read 6 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Read 6 Done Sent Done false 1027 Acked Write 3 Done Sent Done false 1028 Acked Write 7 Done Sent Done false 1029 Acked Read 5 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 0 28 Done 30 2 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:26.763255371Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1520} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 80 73 80 140 132 140 F F F <--- 1 020-039 80 79 80 141 139 141 F F F <--- 2 040-059 80 79 80 141 139 141 T T T <--- 3 060-079 75 75 75 134 134 134 T F T <--- 4 080-099 77 77 77 136 136 136 F F F 5 100-119 78 78 78 138 138 138 F F F 6 120-139 80 75 80 143 134 143 F F F <--- 7 140-159 80 77 80 143 136 143 T F T <--- 8 160-179 78 78 78 137 137 137 F F F 9 180-199 78 78 78 137 137 137 F F F 10 200-219 80 79 80 140 139 140 F F F <--- 11 220-239 80 79 80 140 139 140 F T F <--- 12 240-259 73 73 73 132 132 132 F F F 13 260-279 80 73 80 142 132 142 F F F <--- 14 280-299 80 75 80 142 134 142 F F F <--- 15 300-319 75 75 75 134 134 134 F F F 16 320-339 77 77 77 136 136 136 F F F 17 340-359 77 77 77 136 136 136 F F F 18 360-379 80 79 80 141 139 141 F F F <--- 19 380-399 80 79 80 141 139 141 F F F <--- 20 400-419 80 77 80 140 136 140 F F F <--- 21 420-439 80 78 80 140 137 140 F F F <--- 22 440-459 75 75 75 134 134 134 F F F 23 460-479 77 77 77 136 136 136 F F F 24 480-499 80 75 80 140 134 140 F F F <--- 25 500-519 77 77 77 136 136 136 T F T <--- 26 520-539 80 78 80 140 137 140 F F F <--- 27 540-559 80 76 80 143 135 143 F F F <--- 28 560-579 80 77 80 143 136 143 F F F <--- 29 580-599 77 77 77 136 136 136 F F F Max gen: 80, Max flush: 143 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 40 Verifying data now No change after verify Loop: 40 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 80 80 80 140 140 140 F F F 1 020-039 80 80 80 141 141 141 F F F 2 040-059 81 81 81 144 144 144 F F F 3 060-079 81 81 81 144 144 144 F F F 4 080-099 77 77 77 136 136 136 F F F 5 100-119 78 78 78 138 138 138 F F F 6 120-139 80 80 80 143 143 143 F F F 7 140-159 81 81 81 144 144 144 F F F 8 160-179 78 78 78 137 137 137 F F F 9 180-199 78 78 78 137 137 137 F F F 10 200-219 80 80 80 140 140 140 F F F 11 220-239 80 80 80 140 140 140 F F F 12 240-259 73 73 73 132 132 132 F F F 13 260-279 80 80 80 142 142 142 F F F 14 280-299 80 80 80 142 142 142 F F F 15 300-319 75 75 75 134 134 134 F F F 16 320-339 77 77 77 136 136 136 F F F 17 340-359 77 77 77 136 136 136 F F F 18 360-379 80 80 80 141 141 141 F F F 19 380-399 80 80 80 141 141 141 F F F 20 400-419 80 80 80 140 140 140 F F F 21 420-439 80 80 80 140 140 140 F F F 22 440-459 75 75 75 134 134 134 F F F 23 460-479 77 77 77 136 136 136 F F F 24 480-499 80 80 80 140 140 140 F F F 25 500-519 81 81 81 144 144 144 F F F 26 520-539 80 80 80 140 140 140 F F F 27 540-559 80 80 80 143 143 143 F F F 28 560-579 80 80 80 143 143 143 F F F 29 580-599 77 77 77 136 136 136 F F F Max gen: 81, Max flush: 144 Begin loop 41 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.188040421Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:31.231856521Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.232239773Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.264360126Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.264974229Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.26504807Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.265076216Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.265104492Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528} {"msg":"Crucible 5bccdc71-f1f9-4543-b618-f53b758e7261 has session id: e6771e8b-26c7-4540-973d-f737fa7f71d4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.265643015Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"Upstairs opts: Upstairs UUID: 5bccdc71-f1f9-4543-b618-f53b758e7261, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.265709836Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"Crucible stats registered with UUID: 5bccdc71-f1f9-4543-b618-f53b758e7261","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.265747562Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:31.265774208Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.265969574Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"io task","client":"2","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266011008Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"io task","client":"0","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266031016Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"io task","client":"1","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266176567Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"task":"crutest"} {"msg":"The guest has requested activation with gen:82","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266205514Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528} {"msg":"5bccdc71-f1f9-4543-b618-f53b758e7261 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266269276Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266290793Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"client":"0","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266309131Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"client":"1","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266326229Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"client":"2","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266342836Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"io task","client":"2","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266360304Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"io task","client":"0","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266382662Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"io task","client":"1","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"ds_connection connected from Ok(127.0.0.1:44287)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266478739Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"io task","client":"2","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"ds_connection connected from Ok(127.0.0.1:46583)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266521304Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"io task","client":"1","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"ds_connection connected from Ok(127.0.0.1:37090)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.266638649Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"io task","client":"0","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.267689817Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"client":"1","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.267722682Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"client":"2","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.2677444Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"client":"0","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268288031Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268384789Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"[0]R flush_numbers[0..12]: [140, 141, 144, 144, 136, 138, 143, 144, 137, 137, 140, 140]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268473378Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"downstairs","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"[0]R generation[0..12]: [80, 80, 81, 81, 77, 78, 80, 81, 78, 78, 80, 80]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268497485Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"downstairs","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268518302Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"downstairs","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"[1]R flush_numbers[0..12]: [140, 141, 144, 144, 136, 138, 143, 144, 137, 137, 140, 140]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.26854Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"downstairs","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"[1]R generation[0..12]: [80, 80, 81, 81, 77, 78, 80, 81, 78, 78, 80, 80]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268571366Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"downstairs","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268607041Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"downstairs","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"[2]R flush_numbers[0..12]: [140, 141, 144, 144, 136, 138, 143, 144, 137, 137, 140, 140]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268642797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"[2]R generation[0..12]: [80, 80, 81, 81, 77, 78, 80, 81, 78, 78, 80, 80]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268681992Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"downstairs","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268719387Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"downstairs","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"Max found gen is 82","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268739425Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"downstairs","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"Generation requested: 82 >= found:82","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268766661Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"downstairs","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"Next flush: 145","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268797107Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"downstairs","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268822604Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"":"downstairs","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268843161Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268869258Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"client":"0","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268908443Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"client":"1","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268930621Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"client":"2","":"downstairs","session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268955747Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 } {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.268985983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"5bccdc71-f1f9-4543-b618-f53b758e7261 is now active with session: e6771e8b-26c7-4540-973d-f737fa7f71d4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.26901415Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.269039747Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528,"session_id":"e6771e8b-26c7-4540-973d-f737fa7f71d4"} {"msg":"The guest has finished waiting for activation with:82","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.269066243Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528} {"msg":"Activated sub_volume 5bccdc71-f1f9-4543-b618-f53b758e7261","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:31.26908949Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 418 len 12288 data: 10 10 10 02/30 Write block 305 len 36864 data: 8 7 9 8 8 8 9 8 9 03/30 Read block 30 len 4096 04/30 Read block 580 len 16384 05/30 Flush 06/30 Write block 478 len 28672 data: 7 7 5 6 5 6 6 07/30 Write block 402 len 20480 data: 8 7 7 7 5 08/30 Read block 577 len 16384 09/30 Read block 350 len 4096 10/30 Write block 593 len 28672 data: 5 5 6 7 5 4 2 11/30 Read block 150 len 28672 12/30 Write block 538 len 20480 data: 8 8 8 9 12 13/30 Write block 234 len 24576 data: 5 7 7 7 7 8 14/30 Read block 214 len 40960 15/30 Read block 271 len 28672 16/30 Read block 200 len 32768 17/30 Read block 328 len 16384 18/30 Write block 573 len 8192 data: 2 2 19/30 Write block 253 len 40960 data: 7 8 7 7 6 7 3 3 3 3 20/30 Write block 19 len 36864 data: 3 5 5 8 8 8 9 9 8 21/30 Read block 95 len 4096 22/30 Write block 110 len 32768 data: 7 8 7 9 11 10 16 13 23/30 Read block 364 len 4096 24/30 Read block 118 len 36864 25/30 Read block 4 len 32768 26/30 Read block 3 len 16384 27/30 Write block 200 len 28672 data: 11 12 9 9 10 13 9 28/30 Read block 346 len 12288 29/30 Write block 531 len 32768 data: 8 8 9 9 10 10 8 9 30/30 Write block 298 len 8192 data: 5 6 ---------------------------------------------------------------- Crucible gen:82 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Read 4 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 7 Sent Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Read 1 Sent Done Done false 1009 Acked Write 7 Sent Done Done false 1010 Acked Read 7 Sent Done Done false 1011 Acked Write 5 Sent Done Done false 1012 Acked Write 6 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Read 7 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Write 2 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Write 9 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Write 8 Sent Done Done false 1022 Acked Read 1 Sent Done Done false 1023 Acked Read 9 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Read 3 Sent Done Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 2 2 31 Done 3 28 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1004)) Some(JobId(1004)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:31.288487486Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1528} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 80 80 80 140 140 140 F T T <--- 1 020-039 80 80 80 141 141 141 F T T <--- 2 040-059 81 81 81 144 144 144 F F F 3 060-079 81 81 81 144 144 144 F F F 4 080-099 77 77 77 136 136 136 F F F 5 100-119 78 78 78 138 138 138 F T T <--- 6 120-139 80 80 80 143 143 143 F F F 7 140-159 81 81 81 144 144 144 F F F 8 160-179 78 78 78 137 137 137 F F F 9 180-199 78 78 78 137 137 137 F F F 10 200-219 80 80 80 140 140 140 F T T <--- 11 220-239 80 80 80 140 140 140 F T T <--- 12 240-259 73 73 73 132 132 132 F T T <--- 13 260-279 80 80 80 142 142 142 F T T <--- 14 280-299 80 80 80 142 142 142 F T T <--- 15 300-319 75 82 82 134 145 145 T F F <--- 16 320-339 77 77 77 136 136 136 F F F 17 340-359 77 77 77 136 136 136 F F F 18 360-379 80 80 80 141 141 141 F F F 19 380-399 80 80 80 141 141 141 F F F 20 400-419 80 82 82 140 145 145 T T T <--- 21 420-439 80 82 82 140 145 145 T F F <--- 22 440-459 75 75 75 134 134 134 F F F 23 460-479 77 77 77 136 136 136 F T T <--- 24 480-499 80 80 80 140 140 140 F T T <--- 25 500-519 81 81 81 144 144 144 F F F 26 520-539 80 80 80 140 140 140 F T T <--- 27 540-559 80 80 80 143 143 143 F T T <--- 28 560-579 80 80 80 143 143 143 F T T <--- 29 580-599 77 77 77 136 136 136 F T T <--- Max gen: 82, Max flush: 145 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 41 Verifying data now No change after verify Loop: 41 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 146 146 146 F F F 1 020-039 83 83 83 146 146 146 F F F 2 040-059 81 81 81 144 144 144 F F F 3 060-079 81 81 81 144 144 144 F F F 4 080-099 77 77 77 136 136 136 F F F 5 100-119 83 83 83 146 146 146 F F F 6 120-139 80 80 80 143 143 143 F F F 7 140-159 81 81 81 144 144 144 F F F 8 160-179 78 78 78 137 137 137 F F F 9 180-199 78 78 78 137 137 137 F F F 10 200-219 83 83 83 146 146 146 F F F 11 220-239 83 83 83 146 146 146 F F F 12 240-259 83 83 83 146 146 146 F F F 13 260-279 83 83 83 146 146 146 F F F 14 280-299 83 83 83 146 146 146 F F F 15 300-319 82 82 82 145 145 145 F F F 16 320-339 77 77 77 136 136 136 F F F 17 340-359 77 77 77 136 136 136 F F F 18 360-379 80 80 80 141 141 141 F F F 19 380-399 80 80 80 141 141 141 F F F 20 400-419 83 83 83 146 146 146 F F F 21 420-439 82 82 82 145 145 145 F F F 22 440-459 75 75 75 134 134 134 F F F 23 460-479 83 83 83 146 146 146 F F F 24 480-499 83 83 83 146 146 146 F F F 25 500-519 81 81 81 144 144 144 F F F 26 520-539 83 83 83 146 146 146 F F F 27 540-559 83 83 83 146 146 146 F F F 28 560-579 83 83 83 146 146 146 F F F 29 580-599 83 83 83 146 146 146 F F F Max gen: 83, Max flush: 146 Begin loop 42 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.751832029Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.796935756Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.797550109Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.797612141Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.797632288Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.797659355Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536} {"msg":"Crucible 6e505450-23e3-4ede-b87b-ce32423267b6 has session id: b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.798191378Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"Upstairs opts: Upstairs UUID: 6e505450-23e3-4ede-b87b-ce32423267b6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.798235062Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"Crucible stats registered with UUID: 6e505450-23e3-4ede-b87b-ce32423267b6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.79825374Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:35.798275347Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.798500989Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"io task","client":"1","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.798545033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"io task","client":"0","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.798566981Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"io task","client":"2","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.798796241Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"task":"crutest"} {"msg":"The guest has requested activation with gen:84","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.798830967Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536} {"msg":"6e505450-23e3-4ede-b87b-ce32423267b6 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.798900148Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.798938774Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"client":"0","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.79896676Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"client":"1","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.798994596Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"client":"2","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.799022873Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"io task","client":"2","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.799053489Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"io task","client":"0","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.799081246Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"io task","client":"1","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"ds_connection connected from Ok(127.0.0.1:54364)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.799108472Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"io task","client":"2","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"ds_connection connected from Ok(127.0.0.1:49396)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.799138229Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"io task","client":"1","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"ds_connection connected from Ok(127.0.0.1:61781)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.799292929Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"io task","client":"0","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.80039355Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"client":"2","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.800580877Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"client":"0","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.800782901Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"client":"1","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.80094737Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801094052Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"[0]R flush_numbers[0..12]: [146, 146, 144, 144, 136, 146, 143, 144, 137, 137, 146, 146]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.80134662Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"downstairs","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"[0]R generation[0..12]: [83, 83, 81, 81, 77, 83, 80, 81, 78, 78, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801376966Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"downstairs","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801398494Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"downstairs","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"[1]R flush_numbers[0..12]: [146, 146, 144, 144, 136, 146, 143, 144, 137, 137, 146, 146]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801418021Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"downstairs","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"[1]R generation[0..12]: [83, 83, 81, 81, 77, 83, 80, 81, 78, 78, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801436509Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"downstairs","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801456066Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"downstairs","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"[2]R flush_numbers[0..12]: [146, 146, 144, 144, 136, 146, 143, 144, 137, 137, 146, 146]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801474944Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"downstairs","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"[2]R generation[0..12]: [83, 83, 81, 81, 77, 83, 80, 81, 78, 78, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801493142Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"downstairs","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801512659Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"downstairs","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"Max found gen is 84","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801528837Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"downstairs","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"Generation requested: 84 >= found:84","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-11-06T20:14:35.801545765Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"downstairs","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"Next flush: 147","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801568522Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"downstairs","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.80158491Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"":"downstairs","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801601928Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801617266Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"client":"0","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801636284Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"client":"1","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801653082Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"client":"2","":"downstairs","session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801670019Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801686057Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"6e505450-23e3-4ede-b87b-ce32423267b6 is now active with session: b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801704085Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801719383Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536,"session_id":"b57e2e05-5dd5-46a3-8c23-fbf5b7aaccb5"} {"msg":"The guest has finished waiting for activation with:84","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801735351Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Activated sub_volume 6e505450-23e3-4ede-b87b-ce32423267b6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:35.801751049Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 20 len 12288 data: 6 6 9 02/30 Read block 184 len 8192 03/30 Read block 60 len 24576 04/30 Write block 492 len 40960 data: 6 6 4 4 3 3 5 6 6 8 05/30 Read block 357 len 20480 06/30 Write block 244 len 24576 data: 9 8 7 5 5 5 07/30 Read block 398 len 4096 08/30 Write block 403 len 36864 data: 8 8 8 6 5 6 7 8 8 09/30 Read block 18 len 24576 10/30 Read block 591 len 32768 11/30 Read block 480 len 32768 12/30 Read block 450 len 28672 13/30 Write block 437 len 40960 data: 7 8 11 8 9 8 8 9 8 5 14/30 Write block 443 len 32768 data: 9 10 9 6 5 7 9 11 15/30 Write block 570 len 40960 data: 3 3 2 3 3 3 5 6 5 6 16/30 Read block 471 len 40960 17/30 Write block 296 len 32768 data: 4 5 6 7 6 6 7 7 18/30 Read block 73 len 12288 19/30 Flush 20/30 Write block 35 len 40960 data: 12 13 12 14 12 14 15 13 11 10 21/30 Write block 532 len 36864 data: 9 10 10 11 11 9 10 9 9 22/30 Read block 532 len 12288 23/30 Read block 12 len 12288 24/30 Flush 25/30 Flush 26/30 Write block 467 len 32768 data: 6 5 4 5 6 8 8 7 27/30 Read block 440 len 20480 28/30 Write block 416 len 16384 data: 7 7 11 11 29/30 Read block 403 len 16384 30/30 Write block 192 len 32768 data: 5 4 5 4 6 10 10 10 ---------------------------------------------------------------- Crucible gen:84 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Sent Done false 1001 Acked Read 2 Done Sent Done false 1002 Acked Read 6 Done Sent Done false 1003 Acked Write 10 Done Sent Done false 1004 Acked Read 5 Done Sent Done false 1005 Acked Write 6 Done Sent Done false 1006 Acked Read 1 Done Sent Done false 1007 Acked Write 9 Done Sent Done false 1008 Acked Read 6 Done Sent Done false 1009 Acked Read 8 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Write 8 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Write 8 Done Sent Done false 1017 Acked Read 3 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 10 Done Sent Done false 1020 Acked Write 9 Done Sent Done false 1021 Acked Read 3 Done Sent Done false 1022 Acked Read 3 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 8 Done Sent Done false 1026 Acked Read 5 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 1 32 Done 29 0 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:35.82325849Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1536} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 146 146 146 F F F 1 020-039 84 83 84 148 146 148 F F F <--- 2 040-059 84 81 84 148 144 148 F F F <--- 3 060-079 81 81 81 144 144 144 F F F 4 080-099 77 77 77 136 136 136 F F F 5 100-119 83 83 83 146 146 146 F F F 6 120-139 80 80 80 143 143 143 F F F 7 140-159 81 81 81 144 144 144 F F F 8 160-179 78 78 78 137 137 137 F F F 9 180-199 78 78 78 137 137 137 T F T <--- 10 200-219 83 83 83 146 146 146 F F F 11 220-239 83 83 83 146 146 146 F F F 12 240-259 84 83 84 147 146 147 F F F <--- 13 260-279 83 83 83 146 146 146 F F F 14 280-299 84 83 84 147 146 147 F F F <--- 15 300-319 84 82 84 147 145 147 F F F <--- 16 320-339 77 77 77 136 136 136 F F F 17 340-359 77 77 77 136 136 136 F F F 18 360-379 80 80 80 141 141 141 F F F 19 380-399 80 80 80 141 141 141 F F F 20 400-419 84 83 84 147 146 147 T F T <--- 21 420-439 84 82 84 147 145 147 F F F <--- 22 440-459 84 75 84 147 134 147 F F F <--- 23 460-479 83 83 83 146 146 146 T F T <--- 24 480-499 84 83 84 147 146 147 F F F <--- 25 500-519 84 81 84 147 144 147 F F F <--- 26 520-539 84 83 84 148 146 148 F F F <--- 27 540-559 84 83 84 148 146 148 F F F <--- 28 560-579 84 83 84 147 146 147 F F F <--- 29 580-599 83 83 83 146 146 146 F F F Max gen: 84, Max flush: 148 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 42 Verifying data now No change after verify Loop: 42 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 146 146 146 F F F 1 020-039 84 84 84 148 148 148 F F F 2 040-059 84 84 84 148 148 148 F F F 3 060-079 81 81 81 144 144 144 F F F 4 080-099 77 77 77 136 136 136 F F F 5 100-119 83 83 83 146 146 146 F F F 6 120-139 80 80 80 143 143 143 F F F 7 140-159 81 81 81 144 144 144 F F F 8 160-179 78 78 78 137 137 137 F F F 9 180-199 85 85 85 149 149 149 F F F 10 200-219 83 83 83 146 146 146 F F F 11 220-239 83 83 83 146 146 146 F F F 12 240-259 84 84 84 147 147 147 F F F 13 260-279 83 83 83 146 146 146 F F F 14 280-299 84 84 84 147 147 147 F F F 15 300-319 84 84 84 147 147 147 F F F 16 320-339 77 77 77 136 136 136 F F F 17 340-359 77 77 77 136 136 136 F F F 18 360-379 80 80 80 141 141 141 F F F 19 380-399 80 80 80 141 141 141 F F F 20 400-419 85 85 85 149 149 149 F F F 21 420-439 84 84 84 147 147 147 F F F 22 440-459 84 84 84 147 147 147 F F F 23 460-479 85 85 85 149 149 149 F F F 24 480-499 84 84 84 147 147 147 F F F 25 500-519 84 84 84 147 147 147 F F F 26 520-539 84 84 84 148 148 148 F F F 27 540-559 84 84 84 148 148 148 F F F 28 560-579 84 84 84 147 147 147 F F F 29 580-599 83 83 83 146 146 146 F F F Max gen: 85, Max flush: 149 Begin loop 43 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.239402509Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.284582527Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.285168653Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.285217257Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.285233524Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.285255582Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544} {"msg":"Crucible a721f07d-5d1b-4e46-98de-7c26c0a8910f has session id: 0b5a99fd-9d09-4196-9805-0c336d8914e7","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.285807402Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"Upstairs opts: Upstairs UUID: a721f07d-5d1b-4e46-98de-7c26c0a8910f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.285873174Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"Crucible stats registered with UUID: a721f07d-5d1b-4e46-98de-7c26c0a8910f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.28590506Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:40.285936506Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286136931Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"0","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286179995Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286212231Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"2","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286345864Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"task":"crutest"} {"msg":"The guest has requested activation with gen:86","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286366382Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544} {"msg":"a721f07d-5d1b-4e46-98de-7c26c0a8910f active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286437353Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.2864574Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"client":"0","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286474918Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"client":"1","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286491806Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"client":"2","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286508384Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"2","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286526272Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"0","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286552568Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"ds_connection connected from Ok(127.0.0.1:48068)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286585434Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"2","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"ds_connection connected from Ok(127.0.0.1:53896)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286626739Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"0","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:40.286819565Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:40.28686055Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:40.286911653Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286945459Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.286977885Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.287833967Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"client":"2","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.287930145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"client":"0","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.288338833Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:40.288497303Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.288433291Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"ds_connection connected from Ok(127.0.0.1:36990)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.288793176Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"io task","client":"1","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.290337361Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"client":"1","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"[0]R flush_numbers[0..12]: [146, 148, 148, 144, 136, 146, 143, 144, 137, 149, 146, 146]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.290919778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"downstairs","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"[0]R generation[0..12]: [83, 84, 84, 81, 77, 83, 80, 81, 78, 85, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.290948894Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"downstairs","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.290970121Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"downstairs","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"[1]R flush_numbers[0..12]: [146, 148, 148, 144, 136, 146, 143, 144, 137, 149, 146, 146]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.290989699Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"downstairs","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"[1]R generation[0..12]: [83, 84, 84, 81, 77, 83, 80, 81, 78, 85, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291008207Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"downstairs","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291027834Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"downstairs","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"[2]R flush_numbers[0..12]: [146, 148, 148, 144, 136, 146, 143, 144, 137, 149, 146, 146]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291046752Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"downstairs","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"[2]R generation[0..12]: [83, 84, 84, 81, 77, 83, 80, 81, 78, 85, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291065209Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"downstairs","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291085117Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"downstairs","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"Max found gen is 86","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291101305Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"downstairs","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"Generation requested: 86 >= found:86","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291118283Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"downstairs","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"Next flush: 150","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-11-06T20:14:50.291134391Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"downstairs","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291156888Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"":"downstairs","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291173796Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291189054Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"client":"0","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291209751Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"client":"1","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291226859Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"client":"2","":"downstairs","session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291244757Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291261085Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"a721f07d-5d1b-4e46-98de-7c26c0a8910f is now active with session: 0b5a99fd-9d09-4196-9805-0c336d8914e7","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291278943Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:14:50.291294191Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544,"session_id":"0b5a99fd-9d09-4196-9805-0c336d8914e7"} {"msg":"The guest has finished waiting for activation with:86","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291313048Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544} {"msg":"Activated sub_volume a721f07d-5d1b-4e46-98de-7c26c0a8910f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:50.291327156Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 324 len 32768 data: 7 8 8 7 6 4 4 5 02/30 Write block 315 len 4096 data: 6 03/30 Read block 430 len 8192 04/30 Read block 130 len 28672 05/30 Write block 84 len 28672 data: 6 5 6 6 7 9 8 06/30 Write block 589 len 32768 data: 14 13 9 8 6 6 7 8 07/30 Read block 4 len 24576 08/30 Read block 233 len 20480 09/30 Read block 483 len 16384 10/30 Read block 11 len 40960 11/30 Write block 184 len 8192 data: 9 9 12/30 Read block 297 len 8192 13/30 Read block 366 len 24576 14/30 Flush 15/30 Write block 275 len 28672 data: 7 7 8 8 8 7 7 16/30 Read block 179 len 16384 17/30 Read block 212 len 16384 18/30 Read block 429 len 8192 19/30 Read block 220 len 24576 20/30 Read block 536 len 16384 21/30 Write block 425 len 8192 data: 11 11 22/30 Read block 350 len 32768 23/30 Write block 242 len 8192 data: 9 9 24/30 Read block 460 len 20480 25/30 Read block 498 len 40960 26/30 Write block 122 len 36864 data: 7 8 7 7 6 9 12 15 13 27/30 Write block 247 len 16384 data: 6 6 6 7 28/30 Write block 116 len 20480 data: 17 14 12 12 13 29/30 Write block 266 len 20480 data: 7 9 8 9 10 30/30 Write block 133 len 28672 data: 12 11 9 12 11 11 9 ---------------------------------------------------------------- Crucible gen:86 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 1 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Read 7 Done Done Done false 1004 Acked Write 7 Done Done Done false 1005 Acked Write 8 Done Done Done false 1006 Acked Read 6 Done Done Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 4 Done Sent Done false 1009 Acked Read 10 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Read 2 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 7 Done Sent Done false 1015 Acked Read 4 Done Sent Done false 1016 Acked Read 4 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Write 2 Done Sent Done false 1021 Acked Read 8 Done Sent Done false 1022 Acked Write 2 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Read 10 Done Sent Done false 1025 Acked Write 9 Sent Sent Done false 1026 Acked Write 4 Sent Sent Done false 1027 Acked Write 5 Sent Sent Sent false 1028 Acked Write 5 Sent Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 5 23 3 31 Done 25 7 27 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1013)) None Some(JobId(1013)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:50.309832995Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1544} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 146 146 146 F F F 1 020-039 84 84 84 148 148 148 F F F 2 040-059 84 84 84 148 148 148 F F F 3 060-079 81 81 81 144 144 144 F F F 4 080-099 86 77 86 150 136 150 F T F <--- 5 100-119 83 83 83 146 146 146 T F T <--- 6 120-139 80 80 80 143 143 143 T F T <--- 7 140-159 81 81 81 144 144 144 F F F 8 160-179 78 78 78 137 137 137 F F F 9 180-199 86 85 86 150 149 150 F F F <--- 10 200-219 83 83 83 146 146 146 F F F 11 220-239 83 83 83 146 146 146 F F F 12 240-259 84 84 84 147 147 147 T F T <--- 13 260-279 83 83 83 146 146 146 T F T <--- 14 280-299 84 84 84 147 147 147 T F T <--- 15 300-319 86 84 86 150 147 150 F T F <--- 16 320-339 86 77 86 150 136 150 F T F <--- 17 340-359 77 77 77 136 136 136 F F F 18 360-379 80 80 80 141 141 141 F F F 19 380-399 80 80 80 141 141 141 F F F 20 400-419 85 85 85 149 149 149 F F F 21 420-439 84 84 84 147 147 147 T F T <--- 22 440-459 84 84 84 147 147 147 F F F 23 460-479 85 85 85 149 149 149 F F F 24 480-499 84 84 84 147 147 147 F F F 25 500-519 84 84 84 147 147 147 F F F 26 520-539 84 84 84 148 148 148 F F F 27 540-559 84 84 84 148 148 148 F F F 28 560-579 84 84 84 147 147 147 F F F 29 580-599 86 83 86 150 146 150 F T F <--- Max gen: 86, Max flush: 150 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 43 Verifying data now No change after verify Loop: 43 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 146 146 146 F F F 1 020-039 84 84 84 148 148 148 F F F 2 040-059 84 84 84 148 148 148 F F F 3 060-079 81 81 81 144 144 144 F F F 4 080-099 86 86 86 150 150 150 F F F 5 100-119 87 87 87 151 151 151 F F F 6 120-139 87 87 87 151 151 151 F F F 7 140-159 81 81 81 144 144 144 F F F 8 160-179 78 78 78 137 137 137 F F F 9 180-199 86 86 86 150 150 150 F F F 10 200-219 83 83 83 146 146 146 F F F 11 220-239 83 83 83 146 146 146 F F F 12 240-259 87 87 87 151 151 151 F F F 13 260-279 87 87 87 151 151 151 F F F 14 280-299 87 87 87 151 151 151 F F F 15 300-319 86 86 86 150 150 150 F F F 16 320-339 86 86 86 150 150 150 F F F 17 340-359 77 77 77 136 136 136 F F F 18 360-379 80 80 80 141 141 141 F F F 19 380-399 80 80 80 141 141 141 F F F 20 400-419 85 85 85 149 149 149 F F F 21 420-439 87 87 87 151 151 151 F F F 22 440-459 84 84 84 147 147 147 F F F 23 460-479 85 85 85 149 149 149 F F F 24 480-499 84 84 84 147 147 147 F F F 25 500-519 84 84 84 147 147 147 F F F 26 520-539 84 84 84 148 148 148 F F F 27 540-559 84 84 84 148 148 148 F F F 28 560-579 84 84 84 147 147 147 F F F 29 580-599 86 86 86 150 150 150 F F F Max gen: 87, Max flush: 151 Begin loop 44 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.711665137Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.757704797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.758334328Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.758383751Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.758400339Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.758422627Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552} {"msg":"Crucible 2b9b0935-f5e3-4a90-ad05-073bde2461c3 has session id: 874e12ab-900a-4e28-9c27-f448feca52bb","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759038459Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"Upstairs opts: Upstairs UUID: 2b9b0935-f5e3-4a90-ad05-073bde2461c3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759099901Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"Crucible stats registered with UUID: 2b9b0935-f5e3-4a90-ad05-073bde2461c3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759129917Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:54.759155554Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759389285Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"io task","client":"0","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.75942966Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"io task","client":"1","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759451397Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"io task","client":"2","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759640133Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"task":"crutest"} {"msg":"The guest has requested activation with gen:88","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.75966313Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552} {"msg":"2b9b0935-f5e3-4a90-ad05-073bde2461c3 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759733031Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759752749Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"client":"0","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759769777Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"client":"1","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759787284Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"client":"2","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759816101Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"io task","client":"2","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759840538Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"io task","client":"0","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759864685Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"io task","client":"1","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:44867)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759896771Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"io task","client":"2","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:33711)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.759964532Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"io task","client":"1","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:36390)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.760011636Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"io task","client":"0","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.761221304Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"client":"1","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.761255699Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"client":"2","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.761276317Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"client":"0","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.761818409Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.761848105Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"[0]R flush_numbers[0..12]: [146, 148, 148, 144, 150, 151, 151, 144, 137, 150, 146, 146]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.761877311Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"downstairs","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"[0]R generation[0..12]: [83, 84, 84, 81, 86, 87, 87, 81, 78, 86, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.761897179Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"downstairs","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.761917506Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"downstairs","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"[1]R flush_numbers[0..12]: [146, 148, 148, 144, 150, 151, 151, 144, 137, 150, 146, 146]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.761936714Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"downstairs","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"[1]R generation[0..12]: [83, 84, 84, 81, 86, 87, 87, 81, 78, 86, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.761955111Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"downstairs","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.761974929Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"downstairs","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"[2]R flush_numbers[0..12]: [146, 148, 148, 144, 150, 151, 151, 144, 137, 150, 146, 146]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.761994086Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"downstairs","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"[2]R generation[0..12]: [83, 84, 84, 81, 86, 87, 87, 81, 78, 86, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762015624Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"downstairs","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762035701Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"downstairs","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"Max found gen is 88","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762051909Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"downstairs","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"Generation requested: 88 >= found:88","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762069007Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"downstairs","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"Next flush: 152","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762085325Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"downstairs","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762101583Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"":"downstairs","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {Wait for a query_work_queue command to finish before sending IO "msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762133869Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762150807Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"client":"0","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762167855Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"client":"1","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762184633Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"client":"2","":"downstairs","session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.76220175Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762217988Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"2b9b0935-f5e3-4a90-ad05-073bde2461c3 is now active with session: 874e12ab-900a-4e28-9c27-f448feca52bb","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762236396Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762251564Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552,"session_id":"874e12ab-900a-4e28-9c27-f448feca52bb"} {"msg":"The guest has finished waiting for activation with:88","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.762267552Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552} {"msg":"Activated sub_volume 2b9b0935-f5e3-4a90-ad05-073bde2461c3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:54.76228137Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 449 len 8192 data: 10 12 02/30 Read block 143 len 16384 03/30 Read block 533 len 4096 04/30 Read block 412 len 24576 05/30 Read block 272 len 8192 06/30 Flush 07/30 Write block 170 len 40960 data: 4 4 4 3 6 6 8 8 9 13 08/30 Write block 556 len 36864 data: 7 7 6 5 4 5 5 5 5 09/30 Write block 354 len 32768 data: 6 7 7 8 7 7 5 6 10/30 Read block 457 len 8192 11/30 Write block 409 len 20480 data: 8 9 9 8 7 12/30 Flush 13/30 Write block 254 len 40960 data: 9 8 8 7 8 4 4 4 4 3 14/30 Read block 464 len 32768 15/30 Read block 429 len 36864 16/30 Read block 0 len 32768 17/30 Read block 184 len 28672 18/30 Read block 69 len 24576 19/30 Read block 297 len 28672 20/30 Write block 475 len 16384 data: 8 7 6 8 21/30 Read block 528 len 20480 22/30 Write block 238 len 36864 data: 8 9 8 9 10 10 10 9 8 23/30 Flush 24/30 Write block 531 len 8192 data: 9 10 25/30 Read block 352 len 4096 26/30 Read block 375 len 40960 27/30 Write block 29 len 36864 data: 7 7 7 7 9 11 13 14 13 28/30 Write block 385 len 8192 data: 5 4 29/30 Write block 271 len 36864 data: 11 11 10 8 8 8 9 9 9 30/30 Read block 41 len 28672 ---------------------------------------------------------------- Crucible gen:88 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Read 6 Done Sent Done false 1004 Acked Read 2 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 10 Done Sent Done false 1007 Acked Write 9 Done Sent Done false 1008 Acked Write 8 Done Sent Done false 1009 Acked Read 2 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Read 7 Done Sent Done false 1017 Acked Read 6 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Read 5 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 2 Done Sent Done false 1024 Acked Read 1 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Write 9 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Write 9 Done Sent Done false 1029 Acked Read 7 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 1 28 Done 30 3 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:54.783269166Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1552} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 146 146 146 F F F 1 020-039 84 84 84 148 148 148 T F T <--- 2 040-059 84 84 84 148 148 148 F F F 3 060-079 81 81 81 144 144 144 F F F 4 080-099 86 86 86 150 150 150 F F F 5 100-119 87 87 87 151 151 151 F F F 6 120-139 87 87 87 151 151 151 F F F 7 140-159 81 81 81 144 144 144 F F F 8 160-179 88 78 88 153 137 153 F F F <--- 9 180-199 86 86 86 150 150 150 F F F 10 200-219 83 83 83 146 146 146 F F F 11 220-239 88 83 88 154 146 154 F F F <--- 12 240-259 88 87 88 154 151 154 F F F <--- 13 260-279 88 87 88 154 151 154 T F T <--- 14 280-299 87 87 87 151 151 151 F F F 15 300-319 86 86 86 150 150 150 F F F 16 320-339 86 86 86 150 150 150 F F F 17 340-359 88 77 88 153 136 153 F F F <--- 18 360-379 88 80 88 153 141 153 F F F <--- 19 380-399 80 80 80 141 141 141 T F T <--- 20 400-419 88 85 88 153 149 153 F F F <--- 21 420-439 87 87 87 151 151 151 F F F 22 440-459 88 84 88 152 147 152 F T F <--- 23 460-479 88 85 88 154 149 154 F F F <--- 24 480-499 84 84 84 147 147 147 F F F 25 500-519 84 84 84 147 147 147 F F F 26 520-539 84 84 84 148 148 148 T F T <--- 27 540-559 88 84 88 153 148 153 F F F <--- 28 560-579 88 84 88 153 147 153 F F F <--- 29 580-599 86 86 86 150 150 150 F F F Max gen: 88, Max flush: 154 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 44 Verifying data now No change after verify Loop: 44 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 146 146 146 F F F 1 020-039 89 89 89 155 155 155 F F F 2 040-059 84 84 84 148 148 148 F F F 3 060-079 81 81 81 144 144 144 F F F 4 080-099 86 86 86 150 150 150 F F F 5 100-119 87 87 87 151 151 151 F F F 6 120-139 87 87 87 151 151 151 F F F 7 140-159 81 81 81 144 144 144 F F F 8 160-179 88 88 88 153 153 153 F F F 9 180-199 86 86 86 150 150 150 F F F 10 200-219 83 83 83 146 146 146 F F F 11 220-239 88 88 88 154 154 154 F F F 12 240-259 88 88 88 154 154 154 F F F 13 260-279 89 89 89 155 155 155 F F F 14 280-299 87 87 87 151 151 151 F F F 15 300-319 86 86 86 150 150 150 F F F 16 320-339 86 86 86 150 150 150 F F F 17 340-359 88 88 88 153 153 153 F F F 18 360-379 88 88 88 153 153 153 F F F 19 380-399 89 89 89 155 155 155 F F F 20 400-419 88 88 88 153 153 153 F F F 21 420-439 87 87 87 151 151 151 F F F 22 440-459 88 88 88 152 152 152 F F F 23 460-479 88 88 88 154 154 154 F F F 24 480-499 84 84 84 147 147 147 F F F 25 500-519 84 84 84 147 147 147 F F F 26 520-539 89 89 89 155 155 155 F F F 27 540-559 88 88 88 153 153 153 F F F 28 560-579 88 88 88 153 153 153 F F F 29 580-599 86 86 86 150 150 150 F F F Max gen: 89, Max flush: 155 Begin loop 45 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.190604584Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.235627401Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.23627457Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.236327513Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.236344041Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.236366258Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560} {"msg":"Crucible dccfaa22-2ac5-4955-986d-bf37ffe04a7b has session id: 24e74d31-20c1-4b8c-a822-fe770ea25eaf","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.23690778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"Upstairs opts: Upstairs UUID: dccfaa22-2ac5-4955-986d-bf37ffe04a7b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.236949675Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"Crucible stats registered with UUID: dccfaa22-2ac5-4955-986d-bf37ffe04a7b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.236969192Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:59.23698538Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.237174966Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"io task","client":"1","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.237212352Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"io task","client":"0","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.237242778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"io task","client":"2","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.237509454Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"task":"crutest"} {"msg":"The guest has requested activation with gen:90","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.237536631Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560} {"msg":"dccfaa22-2ac5-4955-986d-bf37ffe04a7b active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.237595473Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.237628109Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"client":"0","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.237660265Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"client":"1","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.237694621Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"client":"2","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.237789919Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"io task","client":"2","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.237831214Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"io task","client":"0","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.237963417Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"io task","client":"1","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"ds_connection connected from Ok(127.0.0.1:53653)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.237994123Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"io task","client":"2","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"ds_connection connected from Ok(127.0.0.1:59828)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.238013141Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"io task","client":"0","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"ds_connection connected from Ok(127.0.0.1:39692)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.238031269Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"io task","client":"1","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.239113622Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"client":"2","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.239154367Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"client":"0","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.239754432Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.239934129Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.240328089Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"client":"1","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"[0]R flush_numbers[0..12]: [146, 155, 148, 144, 150, 151, 151, 144, 153, 150, 146, 154]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.240843994Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"downstairs","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"[0]R generation[0..12]: [83, 89, 84, 81, 86, 87, 87, 81, 88, 86, 83, 88]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.240886219Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"downstairs","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.240920245Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"downstairs","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"[1]R flush_numbers[0..12]: [146, 155, 148, 144, 150, 151, 151, 144, 153, 150, 146, 154]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.24095401Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"downstairs","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"[1]R generation[0..12]: [83, 89, 84, 81, 86, 87, 87, 81, 88, 86, 83, 88]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.240985057Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"downstairs","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241017582Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"downstairs","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"[2]R flush_numbers[0..12]: [146, 155, 148, 144, 150, 151, 151, 144, 153, 150, 146, 154]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241054618Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"downstairs","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"[2]R generation[0..12]: [83, 89, 84, 81, 86, 87, 87, 81, 88, 86, 83, 88]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241091633Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241136727Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"downstairs","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"Max found gen is 90","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241170893Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"downstairs","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"Generation requested: 90 >= found:90","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241203629Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"downstairs","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"Next flush: 156","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:14:59.241235605Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"downstairs","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241271261Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"":"downstairs","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241296257Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241312195Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"client":"0","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241329553Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"client":"1","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241346491Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"client":"2","":"downstairs","session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241363599Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241379727Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"dccfaa22-2ac5-4955-986d-bf37ffe04a7b is now active with session: 24e74d31-20c1-4b8c-a822-fe770ea25eaf","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241397914Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241413003Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560,"session_id":"24e74d31-20c1-4b8c-a822-fe770ea25eaf"} {"msg":"The guest has finished waiting for activation with:90","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.24142893Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560} {"msg":"Activated sub_volume dccfaa22-2ac5-4955-986d-bf37ffe04a7b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:14:59.241442869Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 458 len 24576 data: 8 8 4 4 4 4 02/30 Read block 364 len 24576 03/30 Write block 589 len 32768 data: 15 14 10 9 7 7 8 9 04/30 Write block 551 len 20480 data: 8 7 8 9 8 05/30 Flush 06/30 Write block 178 len 20480 data: 10 14 15 16 15 07/30 Read block 35 len 24576 08/30 Read block 548 len 28672 09/30 Read block 564 len 32768 10/30 Flush 11/30 Write block 266 len 12288 data: 8 10 9 12/30 Read block 368 len 28672 13/30 Flush 14/30 Write block 436 len 28672 data: 8 8 9 12 9 10 9 15/30 Flush 16/30 Flush 17/30 Write block 162 len 12288 data: 6 5 6 18/30 Write block 256 len 12288 data: 9 8 9 19/30 Read block 560 len 28672 20/30 Read block 170 len 12288 21/30 Write block 167 len 20480 data: 5 7 7 5 5 22/30 Write block 505 len 24576 data: 8 7 5 7 6 8 23/30 Read block 25 len 16384 24/30 Write block 436 len 12288 data: 9 9 10 25/30 Flush 26/30 Write block 510 len 28672 data: 9 8 8 8 8 8 8 27/30 Read block 431 len 16384 28/30 Write block 507 len 16384 data: 6 8 7 10 29/30 Read block 284 len 4096 30/30 Read block 121 len 8192 ---------------------------------------------------------------- Crucible gen:90 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Write 8 Done Sent Done false 1003 Acked Write 5 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 5 Done Sent Done false 1006 Acked Read 6 Done Sent Done false 1007 Acked Read 7 Done Sent Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 7 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Write 3 Done Sent Done false 1018 Acked Read 7 Done Sent Done false 1019 Acked Read 3 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Read 4 Done Sent Done false 1023 Acked Write 3 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Read 2 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 28 1 29 Done 30 2 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:14:59.260459943Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1560} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 146 146 146 F F F 1 020-039 89 89 89 155 155 155 F F F 2 040-059 84 84 84 148 148 148 F F F 3 060-079 81 81 81 144 144 144 F F F 4 080-099 86 86 86 150 150 150 F F F 5 100-119 87 87 87 151 151 151 F F F 6 120-139 87 87 87 151 151 151 F F F 7 140-159 81 81 81 144 144 144 F F F 8 160-179 90 88 90 161 153 161 F F F <--- 9 180-199 90 86 90 157 150 157 F F F <--- 10 200-219 83 83 83 146 146 146 F F F 11 220-239 88 88 88 154 154 154 F F F 12 240-259 90 88 90 161 154 161 F F F <--- 13 260-279 90 89 90 158 155 158 F F F <--- 14 280-299 87 87 87 151 151 151 F F F 15 300-319 86 86 86 150 150 150 F F F 16 320-339 86 86 86 150 150 150 F F F 17 340-359 88 88 88 153 153 153 F F F 18 360-379 88 88 88 153 153 153 F F F 19 380-399 89 89 89 155 155 155 F F F 20 400-419 88 88 88 153 153 153 F F F 21 420-439 90 87 90 161 151 161 F F F <--- 22 440-459 90 88 90 159 152 159 F T F <--- 23 460-479 90 88 90 156 154 156 F T F <--- 24 480-499 84 84 84 147 147 147 F F F 25 500-519 90 84 90 161 147 161 T F T <--- 26 520-539 89 89 89 155 155 155 F F F 27 540-559 90 88 90 156 153 156 F F F <--- 28 560-579 88 88 88 153 153 153 F F F 29 580-599 90 86 90 156 150 156 F F F <--- Max gen: 90, Max flush: 161 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 45 Verifying data now No change after verify Loop: 45 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 146 146 146 F F F 1 020-039 89 89 89 155 155 155 F F F 2 040-059 84 84 84 148 148 148 F F F 3 060-079 81 81 81 144 144 144 F F F 4 080-099 86 86 86 150 150 150 F F F 5 100-119 87 87 87 151 151 151 F F F 6 120-139 87 87 87 151 151 151 F F F 7 140-159 81 81 81 144 144 144 F F F 8 160-179 90 90 90 161 161 161 F F F 9 180-199 90 90 90 157 157 157 F F F 10 200-219 83 83 83 146 146 146 F F F 11 220-239 88 88 88 154 154 154 F F F 12 240-259 90 90 90 161 161 161 F F F 13 260-279 90 90 90 158 158 158 F F F 14 280-299 87 87 87 151 151 151 F F F 15 300-319 86 86 86 150 150 150 F F F 16 320-339 86 86 86 150 150 150 F F F 17 340-359 88 88 88 153 153 153 F F F 18 360-379 88 88 88 153 153 153 F F F 19 380-399 89 89 89 155 155 155 F F F 20 400-419 88 88 88 153 153 153 F F F 21 420-439 90 90 90 161 161 161 F F F 22 440-459 90 90 90 159 159 159 F F F 23 460-479 90 90 90 156 156 156 F F F 24 480-499 84 84 84 147 147 147 F F F 25 500-519 91 91 91 162 162 162 F F F 26 520-539 89 89 89 155 155 155 F F F 27 540-559 90 90 90 156 156 156 F F F 28 560-579 88 88 88 153 153 153 F F F 29 580-599 90 90 90 156 156 156 F F F Max gen: 91, Max flush: 162 Begin loop 46 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.650698855Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:03.694517235Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.694912665Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.726449742Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.727078092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.727134015Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.727150803Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.72717719Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568} {"msg":"Crucible 122307bf-3e88-4857-ba2c-5dd8cdf50367 has session id: 3fccb4de-ed87-4efe-afb0-36afeb148784","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.727693715Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"Upstairs opts: Upstairs UUID: 122307bf-3e88-4857-ba2c-5dd8cdf50367, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.727796092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"Crucible stats registered with UUID: 122307bf-3e88-4857-ba2c-5dd8cdf50367","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.727837387Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:03.727860104Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728015004Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"io task","client":"0","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.72805221Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"io task","client":"1","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728084556Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"io task","client":"2","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728248085Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"task":"crutest"} {"msg":"The guest has requested activation with gen:92","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728269512Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568} {"msg":"122307bf-3e88-4857-ba2c-5dd8cdf50367 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728336794Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728356391Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"client":"0","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728373719Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"client":"1","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728390297Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"client":"2","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728416564Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"io task","client":"2","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728454399Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"io task","client":"0","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"ds_connection connected from Ok(127.0.0.1:44924)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728489874Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"io task","client":"2","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"ds_connection connected from Ok(127.0.0.1:41110)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728522461Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"io task","client":"0","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728553826Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"io task","client":"1","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"ds_connection connected from Ok(127.0.0.1:44727)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.728723165Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"io task","client":"1","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.729737607Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"client":"1","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.729772833Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"client":"0","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.72979681Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"client":"2","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730323514Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730354Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"[0]R flush_numbers[0..12]: [146, 155, 148, 144, 150, 151, 151, 144, 161, 157, 146, 154]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730377757Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"downstairs","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"[0]R generation[0..12]: [83, 89, 84, 81, 86, 87, 87, 81, 90, 90, 83, 88]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730397424Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"downstairs","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730417752Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"downstairs","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"[1]R flush_numbers[0..12]: [146, 155, 148, 144, 150, 151, 151, 144, 161, 157, 146, 154]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730437289Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"downstairs","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"[1]R generation[0..12]: [83, 89, 84, 81, 86, 87, 87, 81, 90, 90, 83, 88]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730455787Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"downstairs","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730480284Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"downstairs","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"[2]R flush_numbers[0..12]: [146, 155, 148, 144, 150, 151, 151, 144, 161, 157, 146, 154]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730500271Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"downstairs","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"[2]R generation[0..12]: [83, 89, 84, 81, 86, 87, 87, 81, 90, 90, 83, 88]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730518979Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"downstairs","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730538696Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"downstairs","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"Max found gen is 92","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730555084Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"downstairs","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"Generation requested: 92 >= found:92","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730572042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"downstairs","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msgWait for a query_work_queue command to finish before sending IO ":"Next flush: 163","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730594329Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"downstairs","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730610767Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"":"downstairs","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730627785Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730643113Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"client":"0","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730660121Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"client":"1","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730676829Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"client":"2","":"downstairs","session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730693937Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730710215Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"122307bf-3e88-4857-ba2c-5dd8cdf50367 is now active with session: 3fccb4de-ed87-4efe-afb0-36afeb148784","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730728143Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.73074505Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568,"session_id":"3fccb4de-ed87-4efe-afb0-36afeb148784"} {"msg":"The guest has finished waiting for activation with:92","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730761508Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568} {"msg":"Activated sub_volume 122307bf-3e88-4857-ba2c-5dd8cdf50367","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:03.730775207Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 23 len 24576 data: 9 9 10 10 9 8 02/30 Read block 579 len 36864 03/30 Write block 161 len 36864 data: 6 7 6 7 6 6 6 8 8 04/30 Read block 573 len 24576 05/30 Read block 169 len 36864 06/30 Write block 159 len 16384 data: 7 7 7 8 07/30 Write block 379 len 8192 data: 5 6 08/30 Write block 218 len 8192 data: 7 8 09/30 Read block 224 len 12288 10/30 Write block 361 len 8192 data: 7 5 11/30 Flush 12/30 Write block 91 len 16384 data: 7 6 5 5 13/30 Flush 14/30 Write block 191 len 36864 data: 4 6 5 6 5 7 11 11 11 15/30 Read block 9 len 40960 16/30 Write block 564 len 12288 data: 6 5 5 17/30 Read block 123 len 40960 18/30 Write block 64 len 36864 data: 8 9 8 7 8 7 5 5 5 19/30 Write block 111 len 16384 data: 9 8 10 12 20/30 Write block 464 len 12288 data: 6 6 5 21/30 Read block 403 len 40960 22/30 Read block 23 len 24576 23/30 Flush 24/30 Write block 11 len 28672 data: 5 5 4 4 2 2 2 25/30 Write block 346 len 20480 data: 5 4 5 5 6 26/30 Write block 465 len 24576 data: 7 6 7 6 5 6 27/30 Read block 478 len 8192 28/30 Read block 36 len 4096 29/30 Read block 170 len 8192 30/30 Read block 286 len 28672 ---------------------------------------------------------------- Crucible gen:92 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 9 Sent Done Done false 1002 Acked Write 9 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Read 9 Sent Done Done false 1005 Acked Write 4 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 4 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 9 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Write 3 Sent Done Done false 1016 Acked Read 10 Sent Done Done false 1017 Acked Write 9 Sent Done Done false 1018 Acked Write 4 Sent Done Done false 1019 Acked Write 3 Sent Done Done false 1020 Acked Read 10 Sent Done Done false 1021 Acked Read 6 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Write 5 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Read 7 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 0 30 Done 1 29 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:03.750220617Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1568} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 83 83 83 146 146 146 F T T <--- 1 020-039 89 92 92 155 163 163 T F F <--- 2 040-059 84 84 84 148 148 148 F F F 3 060-079 81 92 92 144 165 165 F F F <--- 4 080-099 86 92 92 150 164 164 F F F <--- 5 100-119 87 92 92 151 165 165 F F F <--- 6 120-139 87 87 87 151 151 151 F F F 7 140-159 81 92 92 144 163 163 F F F <--- 8 160-179 90 92 92 161 163 163 F F F <--- 9 180-199 90 92 92 157 165 165 F F F <--- 10 200-219 83 92 92 146 163 163 F F F <--- 11 220-239 88 88 88 154 154 154 F F F 12 240-259 90 90 90 161 161 161 F F F 13 260-279 90 90 90 158 158 158 F F F 14 280-299 87 87 87 151 151 151 F F F 15 300-319 86 86 86 150 150 150 F F F 16 320-339 86 86 86 150 150 150 F F F 17 340-359 88 88 88 153 153 153 F T T <--- 18 360-379 88 92 92 153 163 163 F F F <--- 19 380-399 89 92 92 155 163 163 F F F <--- 20 400-419 88 88 88 153 153 153 F F F 21 420-439 90 90 90 161 161 161 F F F 22 440-459 90 90 90 159 159 159 F F F 23 460-479 90 92 92 156 165 165 F T T <--- 24 480-499 84 84 84 147 147 147 F F F 25 500-519 91 91 91 162 162 162 F F F 26 520-539 89 89 89 155 155 155 F F F 27 540-559 90 90 90 156 156 156 F F F 28 560-579 88 92 92 153 165 165 F F F <--- 29 580-599 90 90 90 156 156 156 F F F Max gen: 92, Max flush: 165 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 46 Verifying data now No change after verify Loop: 46 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 93 93 93 166 166 166 F F F 1 020-039 92 92 92 163 163 163 F F F 2 040-059 84 84 84 148 148 148 F F F 3 060-079 92 92 92 165 165 165 F F F 4 080-099 92 92 92 164 164 164 F F F 5 100-119 92 92 92 165 165 165 F F F 6 120-139 87 87 87 151 151 151 F F F 7 140-159 92 92 92 163 163 163 F F F 8 160-179 92 92 92 163 163 163 F F F 9 180-199 92 92 92 165 165 165 F F F 10 200-219 92 92 92 163 163 163 F F F 11 220-239 88 88 88 154 154 154 F F F 12 240-259 90 90 90 161 161 161 F F F 13 260-279 90 90 90 158 158 158 F F F 14 280-299 87 87 87 151 151 151 F F F 15 300-319 86 86 86 150 150 150 F F F 16 320-339 86 86 86 150 150 150 F F F 17 340-359 93 93 93 166 166 166 F F F 18 360-379 92 92 92 163 163 163 F F F 19 380-399 92 92 92 163 163 163 F F F 20 400-419 88 88 88 153 153 153 F F F 21 420-439 90 90 90 161 161 161 F F F 22 440-459 90 90 90 159 159 159 F F F 23 460-479 93 93 93 166 166 166 F F F 24 480-499 84 84 84 147 147 147 F F F 25 500-519 91 91 91 162 162 162 F F F 26 520-539 89 89 89 155 155 155 F F F 27 540-559 90 90 90 156 156 156 F F F 28 560-579 92 92 92 165 165 165 F F F 29 580-599 90 90 90 156 156 156 F F F Max gen: 93, Max flush: 166 Begin loop 47 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.189189719Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:08.23307083Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.233461991Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.264494811Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.265052431Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.265100965Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.265117143Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.26513915Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576} {"msg":"Crucible fc730703-db99-4266-a2b3-2dce873819ec has session id: fe8a41fa-39e7-4958-896f-fb0e48cebdb6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.265671323Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"Upstairs opts: Upstairs UUID: fc730703-db99-4266-a2b3-2dce873819ec, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.265712848Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"Crucible stats registered with UUID: fc730703-db99-4266-a2b3-2dce873819ec","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.265743664Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:08.26577189Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.265951058Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"io task","client":"1","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.26601534Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"io task","client":"0","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.266043086Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"io task","client":"2","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.266139364Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"task":"crutest"} {"msg":"The guest has requested activation with gen:94","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.266159372Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576} {"msg":"fc730703-db99-4266-a2b3-2dce873819ec active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.266216634Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.266246581Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"client":"0","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.266266008Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"client":"1","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.266287085Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"client":"2","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.266325771Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"io task","client":"2","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.266360386Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"io task","client":"0","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.266393782Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"io task","client":"1","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"ds_connection connected from Ok(127.0.0.1:56484)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.266615124Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"io task","client":"2","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"ds_connection connected from Ok(127.0.0.1:55277)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.266659629Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"io task","client":"0","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"ds_connection connected from Ok(127.0.0.1:42438)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.266698494Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"io task","client":"1","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.267545117Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"client":"1","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.267633546Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"client":"2","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.267753431Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"client":"0","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268123534Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268202564Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"[0]R flush_numbers[0..12]: [166, 163, 148, 165, 164, 165, 151, 163, 163, 165, 163, 154]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268303262Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"downstairs","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"[0]R generation[0..12]: [93, 92, 84, 92, 92, 92, 87, 92, 92, 92, 92, 88]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268327619Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"downstairs","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268347826Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"downstairs","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"[1]R flush_numbers[0..12]: [166, 163, 148, 165, 164, 165, 151, 163, 163, 165, 163, 154]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268366983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"downstairs","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"[1]R generation[0..12]: [93, 92, 84, 92, 92, 92, 87, 92, 92, 92, 92, 88]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268386491Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"downstairs","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268406249Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"downstairs","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"[2]R flush_numbers[0..12]: [166, 163, 148, 165, 164, 165, 151, 163, 163, 165, 163, 154]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268425196Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"downstairs","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"[2]R generation[0..12]: [93, 92, 84, 92, 92, 92, 87, 92, 92, 92, 92, 88]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268443534Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"downstairs","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268465761Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"downstairs","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"Max found gen is 94","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268490878Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"downstairs","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"Generation requested: 94 >= found:94","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268518564Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"downstairs","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"Next flush: 167","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268539172Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"downstairs","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268559319Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"":"downstairs","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268580427Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268599614Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"client":"0","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268620611Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"client":"1","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268641569Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"client":"2","":"downstairs","session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268662606Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2025-11-06T20:15:08.268682534Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"fc730703-db99-4266-a2b3-2dce873819ec is now active with session: fe8a41fa-39e7-4958-896f-fb0e48cebdb6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268708211Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268727138Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576,"session_id":"fe8a41fa-39e7-4958-896f-fb0e48cebdb6"} {"msg":"The guest has finished waiting for activation with:94","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268747155Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576} {"msg":"Activated sub_volume fc730703-db99-4266-a2b3-2dce873819ec","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:08.268764623Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 350 len 20480 data: 7 7 6 8 7 02/30 Read block 251 len 28672 03/30 Write block 526 len 4096 data: 6 04/30 Read block 523 len 16384 05/30 Write block 364 len 36864 data: 6 6 7 5 5 4 3 3 3 06/30 Write block 466 len 12288 data: 7 8 7 07/30 Write block 480 len 16384 data: 6 7 6 7 08/30 Write block 6 len 24576 data: 4 4 5 4 4 6 09/30 Read block 312 len 36864 10/30 Read block 529 len 12288 11/30 Read block 286 len 28672 12/30 Read block 331 len 24576 13/30 Read block 314 len 4096 14/30 Write block 57 len 8192 data: 4 6 15/30 Read block 48 len 20480 16/30 Write block 506 len 4096 data: 8 17/30 Read block 442 len 16384 18/30 Write block 551 len 20480 data: 9 8 9 10 9 19/30 Write block 289 len 28672 data: 7 7 6 6 7 6 5 20/30 Read block 591 len 8192 21/30 Write block 346 len 32768 data: 6 5 6 6 8 8 7 9 22/30 Write block 439 len 28672 data: 13 10 11 10 10 11 10 23/30 Write block 403 len 8192 data: 9 9 24/30 Read block 121 len 40960 25/30 Write block 349 len 24576 data: 7 9 9 8 10 8 26/30 Write block 448 len 16384 data: 8 11 13 13 27/30 Write block 545 len 4096 data: 9 28/30 Write block 337 len 32768 data: 8 9 9 7 6 5 5 5 29/30 Read block 595 len 12288 30/30 Read block 356 len 36864 ---------------------------------------------------------------- Crucible gen:94 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Write 1 Done Done Done false 1003 Acked Read 4 Done Done Done false 1004 Acked Write 9 Done Done Done false 1005 Acked Write 3 Done Done Done false 1006 Acked Write 4 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Read 3 Sent Done Done false 1010 Acked Read 7 Sent Done Done false 1011 Acked Read 6 Sent Done Done false 1012 Acked Read 1 Sent Done Done false 1013 Acked Write 2 Sent Done Done false 1014 Acked Read 5 Sent Done Done false 1015 Acked Write 1 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Write 7 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Write 2 Sent Done Done false 1023 Acked Read 10 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Write 4 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Write 8 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Read 9 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 1 0 25 Done 6 29 30 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:08.286119547Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1576} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 93 93 93 166 166 166 F T T <--- 1 020-039 92 92 92 163 163 163 F F F 2 040-059 84 84 84 148 148 148 F T T <--- 3 060-079 92 92 92 165 165 165 F F F 4 080-099 92 92 92 164 164 164 F F F 5 100-119 92 92 92 165 165 165 F F F 6 120-139 87 87 87 151 151 151 F F F 7 140-159 92 92 92 163 163 163 F F F 8 160-179 92 92 92 163 163 163 F F F 9 180-199 92 92 92 165 165 165 F F F 10 200-219 92 92 92 163 163 163 F F F 11 220-239 88 88 88 154 154 154 F F F 12 240-259 90 90 90 161 161 161 F F F 13 260-279 90 90 90 158 158 158 F F F 14 280-299 87 87 87 151 151 151 F T T <--- 15 300-319 86 86 86 150 150 150 F F F 16 320-339 86 86 86 150 150 150 F T T <--- 17 340-359 93 93 93 166 166 166 T T T 18 360-379 92 92 92 163 163 163 T T T 19 380-399 92 92 92 163 163 163 F F F 20 400-419 88 88 88 153 153 153 F T T <--- 21 420-439 90 90 90 161 161 161 F T T <--- 22 440-459 90 90 90 159 159 159 F T T <--- 23 460-479 93 93 93 166 166 166 T T T 24 480-499 84 84 84 147 147 147 F T T <--- 25 500-519 91 91 91 162 162 162 F T T <--- 26 520-539 89 89 89 155 155 155 T T T 27 540-559 90 90 90 156 156 156 F T T <--- 28 560-579 92 92 92 165 165 165 F F F 29 580-599 90 90 90 156 156 156 F F F Max gen: 93, Max flush: 166 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 47 Verifying data now Files /tmp/test_live_repair/test_repair_verify.data and /tmp/test_live_repair/test_repair_verify.data.last differ 342a343 > 7, 345,348d345 < 4, < 4, < 4, < 4, 349a347 > 5, 350a349 > 6, 352,354d350 < 5, < 7, < 7, 355a352,354 > 7, > 9, > 9, 356a356,357 > 10, > 8, 359d359 < 7, diff found after verify Loop: 47 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 167 167 167 F F F 1 020-039 92 92 92 163 163 163 F F F 2 040-059 94 94 94 167 167 167 F F F 3 060-079 92 92 92 165 165 165 F F F 4 080-099 92 92 92 164 164 164 F F F 5 100-119 92 92 92 165 165 165 F F F 6 120-139 87 87 87 151 151 151 F F F 7 140-159 92 92 92 163 163 163 F F F 8 160-179 92 92 92 163 163 163 F F F 9 180-199 92 92 92 165 165 165 F F F 10 200-219 92 92 92 163 163 163 F F F 11 220-239 88 88 88 154 154 154 F F F 12 240-259 90 90 90 161 161 161 F F F 13 260-279 90 90 90 158 158 158 F F F 14 280-299 94 94 94 167 167 167 F F F 15 300-319 86 86 86 150 150 150 F F F 16 320-339 94 94 94 167 167 167 F F F 17 340-359 94 94 94 167 167 167 F F F 18 360-379 94 94 94 167 167 167 F F F 19 380-399 92 92 92 163 163 163 F F F 20 400-419 94 94 94 167 167 167 F F F 21 420-439 94 94 94 167 167 167 F F F 22 440-459 94 94 94 167 167 167 F F F 23 460-479 94 94 94 167 167 167 F F F 24 480-499 94 94 94 167 167 167 F F F 25 500-519 94 94 94 167 167 167 F F F 26 520-539 94 94 94 167 167 167 F F F 27 540-559 94 94 94 167 167 167 F F F 28 560-579 92 92 92 165 165 165 F F F 29 580-599 90 90 90 156 156 156 F F F Max gen: 94, Max flush: 167 Begin loop 48 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.747315409Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:12.792110405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.792493027Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.824368391Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.824922021Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.824981863Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.825001551Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.825029797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587} {"msg":"Crucible 04c37d83-373d-40be-bbaa-c33899bc64bb has session id: e1527ee1-4a9b-4389-b936-c622227a5ccc","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.825617183Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"Upstairs opts: Upstairs UUID: 04c37d83-373d-40be-bbaa-c33899bc64bb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.825683405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"Crucible stats registered with UUID: 04c37d83-373d-40be-bbaa-c33899bc64bb","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.82571906Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:12.825746607Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.825933983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"io task","client":"2","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.825973758Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"io task","client":"1","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.825993036Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"io task","client":"0","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.826144097Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"task":"crutest"} {"msg":"The guest has requested activation with gen:96","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.826164704Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587} {"msg":"04c37d83-373d-40be-bbaa-c33899bc64bb active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.826237705Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.826258093Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"client":"0","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.82627531Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"client":"1","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.826292458Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"client":"2","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.826313875Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"io task","client":"2","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.826342372Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"io task","client":"0","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.826361089Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"io task","client":"1","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"ds_connection connected from Ok(127.0.0.1:58415)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.826527508Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"io task","client":"0","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"ds_connection connected from Ok(127.0.0.1:51911)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.826574343Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"io task","client":"2","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"ds_connection connected from Ok(127.0.0.1:50355)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.826607598Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"io task","client":"1","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.827817256Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"client":"1","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.82786491Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"client":"2","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.827907525Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"client":"0","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828451336Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828485042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"[0]R flush_numbers[0..12]: [167, 163, 167, 165, 164, 165, 151, 163, 163, 165, 163, 154]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828506359Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"downstairs","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"[0]R generation[0..12]: [94, 92, 94, 92, 92, 92, 87, 92, 92, 92, 92, 88]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828525887Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"downstairs","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828545754Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"downstairs","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"[1]R flush_numbers[0..12]: [167, 163, 167, 165, 164, 165, 151, 163, 163, 165, 163, 154]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828565042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"downstairs","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"[1]R generation[0..12]: [94, 92, 94, 92, 92, 92, 87, 92, 92, 92, 92, 88]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828583789Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"downstairs","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828603637Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"downstairs","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"[2]R flush_numbers[0..12]: [167, 163, 167, 165, 164, 165, 151, 163, 163, 165, 163, 154]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828622704Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"downstairs","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"[2]R generation[0..12]: [94, 92, 94, 92, 92, 92, 87, 92, 92, 92, 92, 88]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828640902Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"downstairs","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.82866027Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"downstairs","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"Max found gen is 95","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828676528Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"downstairs","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"Generation requested: 96 >= found:95","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828693516Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"downstairs","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"Next flush: 168","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828709564Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"downstairs","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828725582Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"":"downstairs",Wait for a query_work_queue command to finish before sending IO "":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828751708Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828767026Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"client":"0","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828783834Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"client":"1","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828800472Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"client":"2","":"downstairs","session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.8288174Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828833628Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"04c37d83-373d-40be-bbaa-c33899bc64bb is now active with session: e1527ee1-4a9b-4389-b936-c622227a5ccc","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828851566Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828866754Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587,"session_id":"e1527ee1-4a9b-4389-b936-c622227a5ccc"} {"msg":"The guest has finished waiting for activation with:96","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.828882672Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587} {"msg":"Activated sub_volume 04c37d83-373d-40be-bbaa-c33899bc64bb","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:12.82889694Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 182 len 40960 data: 16 15 10 10 9 7 8 5 4 5 02/30 Write block 243 len 36864 data: 11 11 10 9 7 7 7 8 7 03/30 Read block 182 len 4096 04/30 Write block 36 len 12288 data: 15 14 15 05/30 Write block 467 len 28672 data: 9 8 6 7 7 9 9 06/30 Read block 338 len 40960 07/30 Read block 282 len 28672 08/30 Read block 324 len 36864 09/30 Read block 574 len 20480 10/30 Read block 458 len 16384 11/30 Read block 183 len 12288 12/30 Write block 469 len 40960 data: 7 8 8 10 10 8 9 8 7 9 13/30 Write block 309 len 16384 data: 9 9 10 9 14/30 Flush 15/30 Write block 499 len 16384 data: 7 7 9 8 16/30 Read block 301 len 40960 17/30 Write block 32 len 40960 data: 8 10 12 14 16 15 16 13 15 16 18/30 Read block 361 len 4096 19/30 Write block 273 len 12288 data: 11 9 9 20/30 Write block 314 len 24576 data: 9 7 6 6 5 5 21/30 Write block 575 len 28672 data: 4 6 7 6 7 5 5 22/30 Write block 335 len 24576 data: 9 8 9 10 10 7 23/30 Flush 24/30 Write block 290 len 40960 data: 8 7 7 8 7 6 5 6 7 8 25/30 Write block 223 len 32768 data: 9 6 5 4 4 5 6 6 26/30 Write block 38 len 40960 data: 17 14 16 17 14 12 11 12 11 12 27/30 Read block 350 len 20480 28/30 Read block 590 len 16384 29/30 Read block 499 len 40960 30/30 Read block 442 len 40960 ---------------------------------------------------------------- Crucible gen:96 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Write 3 Done Done Done false 1004 Acked Write 7 Done Done Done false 1005 Acked Read 10 Done Done Done false 1006 Acked Read 7 Done Done Done false 1007 Acked Read 9 Done Done Done false 1008 Acked Read 5 Done Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Write 4 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Read 10 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Read 1 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Write 6 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Write 8 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Read 10 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 21 0 0 21 Done 9 30 30 69 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:12.849541859Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1587} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 167 167 167 F F F 1 020-039 92 96 96 163 169 169 T T T <--- 2 040-059 94 96 96 167 169 169 F T T <--- 3 060-079 92 92 92 165 165 165 F F F 4 080-099 92 92 92 164 164 164 F F F 5 100-119 92 92 92 165 165 165 F F F 6 120-139 87 87 87 151 151 151 F F F 7 140-159 92 92 92 163 163 163 F F F 8 160-179 92 92 92 163 163 163 F F F 9 180-199 92 96 96 165 168 168 T F F <--- 10 200-219 92 92 92 163 163 163 F F F 11 220-239 88 88 88 154 154 154 F T T <--- 12 240-259 90 96 96 161 168 168 T F F <--- 13 260-279 90 96 96 158 169 169 F F F <--- 14 280-299 94 94 94 167 167 167 F T T <--- 15 300-319 86 96 96 150 169 169 F F F <--- 16 320-339 94 96 96 167 169 169 F F F <--- 17 340-359 94 96 96 167 169 169 F F F <--- 18 360-379 94 94 94 167 167 167 F F F 19 380-399 92 92 92 163 163 163 F F F 20 400-419 94 94 94 167 167 167 F F F 21 420-439 94 94 94 167 167 167 F F F 22 440-459 94 94 94 167 167 167 F F F 23 460-479 94 96 96 167 168 168 T F F <--- 24 480-499 94 96 96 167 169 169 F F F <--- 25 500-519 94 96 96 167 169 169 F F F <--- 26 520-539 94 94 94 167 167 167 F F F 27 540-559 94 94 94 167 167 167 F F F 28 560-579 92 96 96 165 169 169 F F F <--- 29 580-599 90 96 96 156 169 169 F F F <--- Max gen: 96, Max flush: 169 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 48 Verifying data now No change after verify Loop: 48 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 167 167 167 F F F 1 020-039 97 97 97 170 170 170 F F F 2 040-059 97 97 97 170 170 170 F F F 3 060-079 92 92 92 165 165 165 F F F 4 080-099 92 92 92 164 164 164 F F F 5 100-119 92 92 92 165 165 165 F F F 6 120-139 87 87 87 151 151 151 F F F 7 140-159 92 92 92 163 163 163 F F F 8 160-179 92 92 92 163 163 163 F F F 9 180-199 96 96 96 168 168 168 F F F 10 200-219 92 92 92 163 163 163 F F F 11 220-239 97 97 97 170 170 170 F F F 12 240-259 96 96 96 168 168 168 F F F 13 260-279 96 96 96 169 169 169 F F F 14 280-299 97 97 97 170 170 170 F F F 15 300-319 96 96 96 169 169 169 F F F 16 320-339 96 96 96 169 169 169 F F F 17 340-359 96 96 96 169 169 169 F F F 18 360-379 94 94 94 167 167 167 F F F 19 380-399 92 92 92 163 163 163 F F F 20 400-419 94 94 94 167 167 167 F F F 21 420-439 94 94 94 167 167 167 F F F 22 440-459 94 94 94 167 167 167 F F F 23 460-479 96 96 96 168 168 168 F F F 24 480-499 96 96 96 169 169 169 F F F 25 500-519 96 96 96 169 169 169 F F F 26 520-539 94 94 94 167 167 167 F F F 27 540-559 94 94 94 167 167 167 F F F 28 560-579 96 96 96 169 169 169 F F F 29 580-599 96 96 96 169 169 169 F F F Max gen: 97, Max flush: 170 Begin loop 49 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.295653163Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:17.339501338Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.3398864Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.371276155Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.371839794Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.371898047Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.371922864Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.37195141Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595} {"msg":"Crucible 0e4a8de5-ce01-4d74-90ef-303436029b57 has session id: a48e174a-5eae-47a8-9c10-a7db4a8d57dc","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.372406513Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"Upstairs opts: Upstairs UUID: 0e4a8de5-ce01-4d74-90ef-303436029b57, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.372459656Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"Crucible stats registered with UUID: 0e4a8de5-ce01-4d74-90ef-303436029b57","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.372490912Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:17.372517989Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.372730112Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"io task","client":"1","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.372766727Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"io task","client":"0","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.372786555Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"io task","client":"2","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.372927967Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"task":"crutest"} {"msg":"The guest has requested activation with gen:98","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.372948754Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595} {"msg":"0e4a8de5-ce01-4d74-90ef-303436029b57 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.373021575Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.373042283Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"client":"0","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.373059391Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"client":"1","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.373080268Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"client":"2","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.373097146Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"io task","client":"2","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.373114453Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"io task","client":"0","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.37314096Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"io task","client":"1","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"ds_connection connected from Ok(127.0.0.1:49928)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.373400137Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"io task","client":"2","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"ds_connection connected from Ok(127.0.0.1:41770)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.373444512Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"io task","client":"0","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"ds_connection connected from Ok(127.0.0.1:36281)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.373467989Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"io task","client":"1","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.374563431Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"client":"1","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.374604576Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"client":"0","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.374714942Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"client":"2","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375161975Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375261613Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"[0]R flush_numbers[0..12]: [167, 170, 170, 165, 164, 165, 151, 163, 163, 168, 163, 170]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375359611Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"downstairs","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"[0]R generation[0..12]: [94, 97, 97, 92, 92, 92, 87, 92, 92, 96, 92, 97]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375383238Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"downstairs","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375403275Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"downstairs","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"[1]R flush_numbers[0..12]: [167, 170, 170, 165, 164, 165, 151, 163, 163, 168, 163, 170]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375426892Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"downstairs","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"[1]R generation[0..12]: [94, 97, 97, 92, 92, 92, 87, 92, 92, 96, 92, 97]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.37544572Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"downstairs","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375465557Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"downstairs","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"[2]R flush_numbers[0..12]: [167, 170, 170, 165, 164, 165, 151, 163, 163, 168, 163, 170]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375484565Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"downstairs","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"[2]R generation[0..12]: [94, 97, 97, 92, 92, 92, 87, 92, 92, 96, 92, 97]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375502603Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"downstairs","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.37552207Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"downstairs","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"Max found gen is 98","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375538088Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"downstairs","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"Generation requested: 98 >= found:98","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-11-06T20:15:17.375554996Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"downstairs","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"Next flush: 171","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375576143Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"downstairs","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375592241Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"":"downstairs","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375608989Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375624327Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"client":"0","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375640995Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"client":"1","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375657613Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"client":"2","":"downstairs","session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375674491Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375692149Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"0e4a8de5-ce01-4d74-90ef-303436029b57 is now active with session: a48e174a-5eae-47a8-9c10-a7db4a8d57dc","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375710197Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375725304Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595,"session_id":"a48e174a-5eae-47a8-9c10-a7db4a8d57dc"} {"msg":"The guest has finished waiting for activation with:98","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375741033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595} {"msg":"Activated sub_volume 0e4a8de5-ce01-4d74-90ef-303436029b57","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:17.375755131Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 515 len 12288 data: 9 9 10 02/30 Read block 270 len 8192 03/30 Write block 229 len 32768 data: 7 7 7 7 7 6 8 8 04/30 Write block 501 len 24576 data: 10 9 9 7 9 9 05/30 Read block 317 len 32768 06/30 Read block 389 len 12288 07/30 Write block 108 len 20480 data: 6 6 8 10 9 08/30 Read block 314 len 4096 09/30 Read block 139 len 32768 10/30 Flush 11/30 Write block 518 len 40960 data: 8 7 6 7 5 4 5 7 7 7 12/30 Flush 13/30 Write block 436 len 8192 data: 10 10 14/30 Read block 208 len 8192 15/30 Read block 252 len 12288 16/30 Read block 413 len 16384 17/30 Write block 18 len 32768 data: 2 4 7 7 10 10 10 11 18/30 Write block 245 len 24576 data: 11 10 8 8 8 9 19/30 Read block 89 len 8192 20/30 Read block 343 len 8192 21/30 Write block 180 len 36864 data: 16 17 17 16 11 11 10 8 9 22/30 Write block 29 len 16384 data: 8 8 8 9 23/30 Write block 254 len 8192 data: 10 9 24/30 Read block 375 len 20480 25/30 Write block 186 len 16384 data: 11 9 10 6 26/30 Read block 491 len 40960 27/30 Read block 503 len 40960 28/30 Write block 37 len 20480 data: 16 18 15 17 18 29/30 Read block 523 len 8192 30/30 Write block 123 len 20480 data: 9 8 8 7 10 ---------------------------------------------------------------- Crucible gen:98 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 2 Sent Done Done false 1002 Acked Write 8 Sent Done Done false 1003 Acked Write 6 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Read 3 Sent Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Read 1 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 10 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Read 4 Sent Done Done false 1016 Acked Write 8 Sent Done Done false 1017 Acked Write 6 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Write 2 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Write 5 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 1 31 Done 1 29 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1011)) Some(JobId(1011)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:17.394959581Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1595} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 167 167 167 F T T <--- 1 020-039 97 97 97 170 170 170 F T T <--- 2 040-059 97 97 97 170 170 170 F T T <--- 3 060-079 92 92 92 165 165 165 F F F 4 080-099 92 92 92 164 164 164 F F F 5 100-119 92 98 98 165 171 171 F F F <--- 6 120-139 87 87 87 151 151 151 F T T <--- 7 140-159 92 92 92 163 163 163 F F F 8 160-179 92 92 92 163 163 163 F F F 9 180-199 96 96 96 168 168 168 F T T <--- 10 200-219 92 92 92 163 163 163 F F F 11 220-239 97 98 98 170 171 171 F F F <--- 12 240-259 96 96 96 168 168 168 F T T <--- 13 260-279 96 96 96 169 169 169 F F F 14 280-299 97 97 97 170 170 170 F F F 15 300-319 96 96 96 169 169 169 F F F 16 320-339 96 96 96 169 169 169 F F F 17 340-359 96 96 96 169 169 169 F F F 18 360-379 94 94 94 167 167 167 F F F 19 380-399 92 92 92 163 163 163 F F F 20 400-419 94 94 94 167 167 167 F F F 21 420-439 94 94 94 167 167 167 F T T <--- 22 440-459 94 94 94 167 167 167 F F F 23 460-479 96 96 96 168 168 168 F F F 24 480-499 96 96 96 169 169 169 F F F 25 500-519 96 98 98 169 172 172 T F F <--- 26 520-539 94 98 98 167 172 172 F F F <--- 27 540-559 94 94 94 167 167 167 F F F 28 560-579 96 96 96 169 169 169 F F F 29 580-599 96 96 96 169 169 169 F F F Max gen: 98, Max flush: 172 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 49 Verifying data now No change after verify Loop: 49 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 99 99 99 173 173 173 F F F 1 020-039 99 99 99 173 173 173 F F F 2 040-059 99 99 99 173 173 173 F F F 3 060-079 92 92 92 165 165 165 F F F 4 080-099 92 92 92 164 164 164 F F F 5 100-119 98 98 98 171 171 171 F F F 6 120-139 99 99 99 173 173 173 F F F 7 140-159 92 92 92 163 163 163 F F F 8 160-179 92 92 92 163 163 163 F F F 9 180-199 99 99 99 173 173 173 F F F 10 200-219 92 92 92 163 163 163 F F F 11 220-239 98 98 98 171 171 171 F F F 12 240-259 99 99 99 173 173 173 F F F 13 260-279 96 96 96 169 169 169 F F F 14 280-299 97 97 97 170 170 170 F F F 15 300-319 96 96 96 169 169 169 F F F 16 320-339 96 96 96 169 169 169 F F F 17 340-359 96 96 96 169 169 169 F F F 18 360-379 94 94 94 167 167 167 F F F 19 380-399 92 92 92 163 163 163 F F F 20 400-419 94 94 94 167 167 167 F F F 21 420-439 99 99 99 173 173 173 F F F 22 440-459 94 94 94 167 167 167 F F F 23 460-479 96 96 96 168 168 168 F F F 24 480-499 96 96 96 169 169 169 F F F 25 500-519 98 98 98 172 172 172 F F F 26 520-539 98 98 98 172 172 172 F F F 27 540-559 94 94 94 167 167 167 F F F 28 560-579 96 96 96 169 169 169 F F F 29 580-599 96 96 96 169 169 169 F F F Max gen: 99, Max flush: 173 Begin loop 50 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.822510821Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:21.866980459Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.867385648Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.899080374Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.899664531Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.899711385Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.899728453Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.89975092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603} {"msg":"Crucible 85e03def-3f1b-4e45-80ec-3be666ee75b8 has session id: c0a3db49-a7ec-4ee1-b6ac-c35c378c110f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.900256596Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"Upstairs opts: Upstairs UUID: 85e03def-3f1b-4e45-80ec-3be666ee75b8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.900299601Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"Crucible stats registered with UUID: 85e03def-3f1b-4e45-80ec-3be666ee75b8","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.900324338Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:21.900341126Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.900589494Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"io task","client":"0","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.900627159Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"io task","client":"2","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.900646807Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"io task","client":"1","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.900801118Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"task":"crutest"} {"msg":"The guest has requested activation with gen:100","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.900829784Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603} {"msg":"85e03def-3f1b-4e45-80ec-3be666ee75b8 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.900904225Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.900924912Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"client":"0","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.90094249Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"client":"1","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.900959578Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"client":"2","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.900976126Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"io task","client":"2","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.901008362Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"io task","client":"0","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.901033568Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"io task","client":"1","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"ds_connection connected from Ok(127.0.0.1:52349)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.901064704Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"io task","client":"2","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"ds_connection connected from Ok(127.0.0.1:60220)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.901096651Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"io task","client":"0","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"ds_connection connected from Ok(127.0.0.1:62987)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.901169591Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"io task","client":"1","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.902311887Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"client":"1","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.902346423Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"client":"2","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.90236878Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"client":"0","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.902910642Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.902939708Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"[0]R flush_numbers[0..12]: [173, 173, 173, 165, 164, 171, 173, 163, 163, 173, 163, 171]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.902958416Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"downstairs","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"[0]R generation[0..12]: [99, 99, 99, 92, 92, 98, 99, 92, 92, 99, 92, 98]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.902977623Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"downstairs","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.902997501Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"downstairs","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"[1]R flush_numbers[0..12]: [173, 173, 173, 165, 164, 171, 173, 163, 163, 173, 163, 171]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903016638Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"downstairs","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"[1]R generation[0..12]: [99, 99, 99, 92, 92, 98, 99, 92, 92, 99, 92, 98]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903035076Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"downstairs","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903054804Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"downstairs","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"[2]R flush_numbers[0..12]: [173, 173, 173, 165, 164, 171, 173, 163, 163, 173, 163, 171]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903073761Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"downstairs","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"[2]R generation[0..12]: [99, 99, 99, 92, 92, 98, 99, 92, 92, 99, 92, 98]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903095679Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"downstairs","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903115596Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"downstairs","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"Max found gen is 100","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903131854Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"downstairs","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"Generation requested: 100 >= found:100","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903148792Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"downstairs","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"Next flush: 174Wait for a query_work_queue command to finish before sending IO ","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903170989Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"downstairs","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903187257Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"":"downstairs","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903204075Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903220393Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"client":"0","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903237081Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"client":"1","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903253689Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"client":"2","":"downstairs","session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903270586Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903286544Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"85e03def-3f1b-4e45-80ec-3be666ee75b8 is now active with session: c0a3db49-a7ec-4ee1-b6ac-c35c378c110f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903304462Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.90331959Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603,"session_id":"c0a3db49-a7ec-4ee1-b6ac-c35c378c110f"} {"msg":"The guest has finished waiting for activation with:100","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903336088Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603} {"msg":"Activated sub_volume 85e03def-3f1b-4e45-80ec-3be666ee75b8","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:21.903350187Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 97 len 24576 data: 7 6 7 6 6 8 02/30 Write block 242 len 32768 data: 11 12 12 12 11 9 9 9 03/30 Write block 188 len 40960 data: 11 7 5 6 7 6 7 6 8 12 04/30 Write block 429 len 40960 data: 8 8 6 5 5 8 6 11 11 11 05/30 Read block 320 len 24576 06/30 Write block 227 len 12288 data: 5 6 8 07/30 Read block 572 len 4096 08/30 Read block 434 len 8192 09/30 Write block 314 len 36864 data: 10 8 7 7 6 6 6 8 6 10/30 Read block 568 len 8192 11/30 Flush 12/30 Write block 520 len 4096 data: 7 13/30 Write block 418 len 12288 data: 12 12 11 14/30 Read block 94 len 28672 15/30 Write block 94 len 16384 data: 6 7 7 8 16/30 Write block 473 len 12288 data: 11 9 10 17/30 Read block 95 len 36864 18/30 Read block 15 len 32768 19/30 Read block 130 len 8192 20/30 Write block 212 len 16384 data: 4 5 6 5 21/30 Write block 227 len 28672 data: 6 7 9 8 8 8 8 22/30 Write block 406 len 12288 data: 7 6 7 23/30 Read block 92 len 36864 24/30 Read block 458 len 36864 25/30 Write block 257 len 8192 data: 9 10 26/30 Read block 391 len 24576 27/30 Read block 101 len 28672 28/30 Write block 458 len 4096 data: 9 29/30 Write block 471 len 8192 data: 9 11 30/30 Read block 235 len 16384 ---------------------------------------------------------------- Crucible gen:100 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Write 10 Done Done Done false 1003 Acked Write 10 Done Done Done false 1004 Acked Read 6 Done Done Done false 1005 Acked Write 3 Done Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Write 9 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 1 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Write 3 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Write 4 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Read 9 Sent Done Done false 1024 Acked Write 2 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Read 7 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Write 2 Sent Done Done false 1029 Acked Read 4 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 1 0 25 Done 6 29 30 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1010)) Some(JobId(1010)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:21.921082242Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1603} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 99 99 99 173 173 173 F F F 1 020-039 99 99 99 173 173 173 F F F 2 040-059 99 99 99 173 173 173 F F F 3 060-079 92 92 92 165 165 165 F F F 4 080-099 92 100 100 164 174 174 T T T <--- 5 100-119 98 100 100 171 174 174 T F F <--- 6 120-139 99 99 99 173 173 173 F F F 7 140-159 92 92 92 163 163 163 F F F 8 160-179 92 92 92 163 163 163 F F F 9 180-199 99 100 100 173 174 174 T F F <--- 10 200-219 92 92 92 163 163 163 F T T <--- 11 220-239 98 100 100 171 174 174 T T T <--- 12 240-259 99 100 100 173 174 174 T T T <--- 13 260-279 96 96 96 169 169 169 F F F 14 280-299 97 97 97 170 170 170 F F F 15 300-319 96 100 100 169 174 174 F F F <--- 16 320-339 96 100 100 169 174 174 F F F <--- 17 340-359 96 96 96 169 169 169 F F F 18 360-379 94 94 94 167 167 167 F F F 19 380-399 92 92 92 163 163 163 F F F 20 400-419 94 94 94 167 167 167 F T T <--- 21 420-439 99 100 100 173 174 174 T T T <--- 22 440-459 94 94 94 167 167 167 F T T <--- 23 460-479 96 96 96 168 168 168 F T T <--- 24 480-499 96 96 96 169 169 169 F F F 25 500-519 98 98 98 172 172 172 F F F 26 520-539 98 98 98 172 172 172 F T T <--- 27 540-559 94 94 94 167 167 167 F F F 28 560-579 96 96 96 169 169 169 F F F 29 580-599 96 96 96 169 169 169 F F F Max gen: 100, Max flush: 174 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 50 Verifying data now No change after verify Loop: 50 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 99 99 99 173 173 173 F F F 1 020-039 99 99 99 173 173 173 F F F 2 040-059 99 99 99 173 173 173 F F F 3 060-079 92 92 92 165 165 165 F F F 4 080-099 101 101 101 175 175 175 F F F 5 100-119 100 100 100 174 174 174 F F F 6 120-139 99 99 99 173 173 173 F F F 7 140-159 92 92 92 163 163 163 F F F 8 160-179 92 92 92 163 163 163 F F F 9 180-199 100 100 100 174 174 174 F F F 10 200-219 101 101 101 175 175 175 F F F 11 220-239 101 101 101 175 175 175 F F F 12 240-259 101 101 101 175 175 175 F F F 13 260-279 96 96 96 169 169 169 F F F 14 280-299 97 97 97 170 170 170 F F F 15 300-319 100 100 100 174 174 174 F F F 16 320-339 100 100 100 174 174 174 F F F 17 340-359 96 96 96 169 169 169 F F F 18 360-379 94 94 94 167 167 167 F F F 19 380-399 92 92 92 163 163 163 F F F 20 400-419 101 101 101 175 175 175 F F F 21 420-439 101 101 101 175 175 175 F F F 22 440-459 101 101 101 175 175 175 F F F 23 460-479 101 101 101 175 175 175 F F F 24 480-499 96 96 96 169 169 169 F F F 25 500-519 98 98 98 172 172 172 F F F 26 520-539 101 101 101 175 175 175 F F F 27 540-559 94 94 94 167 167 167 F F F 28 560-579 96 96 96 169 169 169 F F F 29 580-599 96 96 96 169 169 169 F F F Max gen: 101, Max flush: 175 Begin loop 51 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.364399108Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.410194188Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.410805901Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.410864383Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.410884091Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.410911537Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611} {"msg":"Crucible 6b2fbe35-807c-4fa0-8c26-1f783ba39bb7 has session id: 49c17d57-1c42-4ef4-9a58-4325d0727631","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.411457359Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"Upstairs opts: Upstairs UUID: 6b2fbe35-807c-4fa0-8c26-1f783ba39bb7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.411513642Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"Crucible stats registered with UUID: 6b2fbe35-807c-4fa0-8c26-1f783ba39bb7","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.411543418Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:26.411569775Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.41176411Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"io task","client":"1","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.411814714Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"io task","client":"0","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.41184691Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"io task","client":"2","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.412113456Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"task":"crutest"} {"msg":"The guest has requested activation with gen:102","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.412153661Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611} {"msg":"6b2fbe35-807c-4fa0-8c26-1f783ba39bb7 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.412212664Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.41224291Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"client":"0","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.412262347Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"client":"1","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.412279655Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"client":"2","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.412296763Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"io task","client":"2","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.412314471Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"io task","client":"0","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.412334018Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"io task","client":"1","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"ds_connection connected from Ok(127.0.0.1:46212)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.412368564Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"io task","client":"2","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"ds_connection connected from Ok(127.0.0.1:48905)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.412418918Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"io task","client":"0","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"ds_connection connected from Ok(127.0.0.1:64611)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.412454763Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"io task","client":"1","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.413543076Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"client":"0","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.41366575Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"client":"2","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.414115444Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.414190175Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.414666024Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"client":"1","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"[0]R flush_numbers[0..12]: [173, 173, 173, 165, 175, 174, 173, 163, 163, 174, 175, 175]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415162602Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"downstairs","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"[0]R generation[0..12]: [99, 99, 99, 92, 101, 100, 99, 92, 92, 100, 101, 101]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415191438Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"downstairs","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415213246Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"downstairs","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"[1]R flush_numbers[0..12]: [173, 173, 173, 165, 175, 174, 173, 163, 163, 174, 175, 175]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415232973Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"downstairs","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"[1]R generation[0..12]: [99, 99, 99, 92, 101, 100, 99, 92, 92, 100, 101, 101]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415251831Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"downstairs","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415271658Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"downstairs","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"[2]R flush_numbers[0..12]: [173, 173, 173, 165, 175, 174, 173, 163, 163, 174, 175, 175]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415290816Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"downstairs","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"[2]R generation[0..12]: [99, 99, 99, 92, 101, 100, 99, 92, 92, 100, 101, 101]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415309503Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"downstairs","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415329451Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"downstairs","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"Max found gen is 102","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415346289Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"Generation requested: 102 >= found:102","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415370006Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"downstairs","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"Next flush: 176","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415386024Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"downstairs","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415402122Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"":"downstairs","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415420589Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415439567Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"client":"0","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415456855Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"client":"1","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415473713Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"client":"2","":"downstairs","session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415491151Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 crucible","level":30,"time":"2025-11-06T20:15:26.415510848Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"6b2fbe35-807c-4fa0-8c26-1f783ba39bb7 is now active with session: 49c17d57-1c42-4ef4-9a58-4325d0727631","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415529366Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415544624Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611,"session_id":"49c17d57-1c42-4ef4-9a58-4325d0727631"} {"msg":"The guest has finished waiting for activation with:102","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.415560702Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611} {"msg":"Activated sub_volume 6b2fbe35-807c-4fa0-8c26-1f783ba39bb7","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:26.41557448Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 9 len 20480 data: 5 5 7 6 5 02/30 Read block 337 len 28672 03/30 Read block 171 len 20480 04/30 Read block 565 len 16384 05/30 Write block 28 len 32768 data: 9 9 9 9 10 11 13 15 06/30 Read block 546 len 28672 07/30 Write block 183 len 20480 data: 17 12 12 12 10 08/30 Read block 351 len 20480 09/30 Read block 552 len 8192 10/30 Read block 338 len 32768 11/30 Read block 129 len 24576 12/30 Write block 444 len 28672 data: 12 11 7 6 9 12 14 13/30 Read block 193 len 24576 14/30 Flush 15/30 Write block 35 len 32768 data: 16 17 17 19 16 18 19 15 16/30 Read block 59 len 8192 17/30 Write block 145 len 20480 data: 7 4 4 4 4 18/30 Read block 73 len 28672 19/30 Write block 218 len 16384 data: 8 9 9 7 20/30 Read block 21 len 8192 21/30 Read block 404 len 20480 22/30 Flush 23/30 Write block 389 len 4096 data: 7 24/30 Read block 430 len 4096 25/30 Write block 78 len 36864 data: 8 8 9 8 8 7 7 6 7 26/30 Read block 98 len 16384 27/30 Read block 131 len 32768 28/30 Write block 230 len 28672 data: 9 9 9 9 7 9 9 29/30 Read block 421 len 36864 30/30 Write block 417 len 8192 data: 8 13 ---------------------------------------------------------------- Crucible gen:102 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 5 Done Sent Done false 1003 Acked Read 4 Done Sent Done false 1004 Acked Write 8 Done Sent Done false 1005 Acked Read 7 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Read 8 Done Sent Done false 1010 Acked Read 6 Done Sent Done false 1011 Acked Write 7 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 8 Done Sent Done false 1015 Acked Read 2 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Write 4 Done Sent Done false 1019 Acked Read 2 Done Sent Done false 1020 Acked Read 5 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 1 Done Sent Done false 1023 Acked Read 1 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Read 4 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Write 7 Done Sent Done false 1028 Acked Read 9 Done Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 28 2 31 Done 29 2 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:26.435828858Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1611} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 99 102 176 173 176 F T F <--- 1 020-039 102 99 102 177 173 177 F F F <--- 2 040-059 102 99 102 177 173 177 F F F <--- 3 060-079 92 92 92 165 165 165 T F T <--- 4 080-099 101 101 101 175 175 175 T F T <--- 5 100-119 100 100 100 174 174 174 F F F 6 120-139 99 99 99 173 173 173 F F F 7 140-159 102 92 102 177 163 177 F F F <--- 8 160-179 92 92 92 163 163 163 F F F 9 180-199 102 100 102 176 174 176 F F F <--- 10 200-219 102 101 102 177 175 177 F F F <--- 11 220-239 102 101 102 177 175 177 T F T <--- 12 240-259 101 101 101 175 175 175 F F F 13 260-279 96 96 96 169 169 169 F F F 14 280-299 97 97 97 170 170 170 F F F 15 300-319 100 100 100 174 174 174 F F F 16 320-339 100 100 100 174 174 174 F F F 17 340-359 96 96 96 169 169 169 F F F 18 360-379 94 94 94 167 167 167 F F F 19 380-399 92 92 92 163 163 163 T F T <--- 20 400-419 101 101 101 175 175 175 T F T <--- 21 420-439 101 101 101 175 175 175 F F F 22 440-459 102 101 102 176 175 176 F F F <--- 23 460-479 101 101 101 175 175 175 F F F 24 480-499 96 96 96 169 169 169 F F F 25 500-519 98 98 98 172 172 172 F F F 26 520-539 101 101 101 175 175 175 F F F 27 540-559 94 94 94 167 167 167 F F F 28 560-579 96 96 96 169 169 169 F F F 29 580-599 96 96 96 169 169 169 F F F Max gen: 102, Max flush: 177 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 51 Verifying data now No change after verify Loop: 51 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 176 176 176 F F F 1 020-039 102 102 102 177 177 177 F F F 2 040-059 102 102 102 177 177 177 F F F 3 060-079 103 103 103 178 178 178 F F F 4 080-099 103 103 103 178 178 178 F F F 5 100-119 100 100 100 174 174 174 F F F 6 120-139 99 99 99 173 173 173 F F F 7 140-159 102 102 102 177 177 177 F F F 8 160-179 92 92 92 163 163 163 F F F 9 180-199 102 102 102 176 176 176 F F F 10 200-219 102 102 102 177 177 177 F F F 11 220-239 103 103 103 178 178 178 F F F 12 240-259 101 101 101 175 175 175 F F F 13 260-279 96 96 96 169 169 169 F F F 14 280-299 97 97 97 170 170 170 F F F 15 300-319 100 100 100 174 174 174 F F F 16 320-339 100 100 100 174 174 174 F F F 17 340-359 96 96 96 169 169 169 F F F 18 360-379 94 94 94 167 167 167 F F F 19 380-399 103 103 103 178 178 178 F F F 20 400-419 103 103 103 178 178 178 F F F 21 420-439 101 101 101 175 175 175 F F F 22 440-459 102 102 102 176 176 176 F F F 23 460-479 101 101 101 175 175 175 F F F 24 480-499 96 96 96 169 169 169 F F F 25 500-519 98 98 98 172 172 172 F F F 26 520-539 101 101 101 175 175 175 F F F 27 540-559 94 94 94 167 167 167 F F F 28 560-579 96 96 96 169 169 169 F F F 29 580-599 96 96 96 169 169 169 F F F Max gen: 103, Max flush: 178 Begin loop 52 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.856336077Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.90155864Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.902204228Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.902253732Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.90227198Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.902304776Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619} {"msg":"Crucible 077e753d-567d-42d7-b649-dbc04df24fee has session id: 58bed31b-03be-419b-8309-fdd19a252e23","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.90274659Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"Upstairs opts: Upstairs UUID: 077e753d-567d-42d7-b649-dbc04df24fee, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.902806543Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"Crucible stats registered with UUID: 077e753d-567d-42d7-b649-dbc04df24fee","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.902836609Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:30.902862025Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903118233Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"io task","client":"1","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903167307Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"io task","client":"0","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903192364Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"io task","client":"2","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903359453Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"task":"crutest"} {"msg":"The guest has requested activation with gen:104","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903388669Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619} {"msg":"077e753d-567d-42d7-b649-dbc04df24fee active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903443702Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903472828Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"client":"0","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903491226Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"client":"1","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903514473Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"client":"2","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.90354487Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"io task","client":"2","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903578725Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"io task","client":"0","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903611051Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"io task","client":"1","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"ds_connection connected from Ok(127.0.0.1:33717)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903649086Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"io task","client":"0","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"ds_connection connected from Ok(127.0.0.1:56831)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903683352Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"io task","client":"2","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"ds_connection connected from Ok(127.0.0.1:55259)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.903716978Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"io task","client":"1","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.904899489Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"client":"1","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.904942323Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"client":"0","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.905534029Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.905563955Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.905775138Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"client":"2","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"[0]R flush_numbers[0..12]: [176, 177, 177, 178, 178, 174, 173, 177, 163, 176, 177, 178]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906303022Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"downstairs","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"[0]R generation[0..12]: [102, 102, 102, 103, 103, 100, 99, 102, 92, 102, 102, 103]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906335898Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"downstairs","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906357205Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"downstairs","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"[1]R flush_numbers[0..12]: [176, 177, 177, 178, 178, 174, 173, 177, 163, 176, 177, 178]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906376873Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"downstairs","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"[1]R generation[0..12]: [102, 102, 102, 103, 103, 100, 99, 102, 92, 102, 102, 103]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.90639571Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"downstairs","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906415618Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"downstairs","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"[2]R flush_numbers[0..12]: [176, 177, 177, 178, 178, 174, 173, 177, 163, 176, 177, 178]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906438655Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"downstairs","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"[2]R generation[0..12]: [102, 102, 102, 103, 103, 100, 99, 102, 92, 102, 102, 103]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906458252Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"downstairs","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.90647793Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"downstairs","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"Max found gen is 104","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906494468Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"downstairs","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"Generation requested: 104 >= found:104","v":0,Wait for a query_work_queue command to finish before sending IO "name":"crucible","level":30,"time":"2025-11-06T20:15:30.906517665Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"downstairs","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"Next flush: 179","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906533883Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"downstairs","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906549771Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"":"downstairs","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906566789Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906581877Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"client":"0","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906598705Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"client":"1","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906615423Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"client":"2","":"downstairs","session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906632621Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906648588Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"077e753d-567d-42d7-b649-dbc04df24fee is now active with session: 58bed31b-03be-419b-8309-fdd19a252e23","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906669656Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906685084Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619,"session_id":"58bed31b-03be-419b-8309-fdd19a252e23"} {"msg":"The guest has finished waiting for activation with:104","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.906703292Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619} {"msg":"Activated sub_volume 077e753d-567d-42d7-b649-dbc04df24fee","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:30.90671729Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 365 len 40960 data: 7 8 6 6 5 4 4 4 3 4 02/30 Read block 527 len 28672 03/30 Write block 390 len 24576 data: 7 8 9 8 7 6 04/30 Read block 383 len 4096 05/30 Flush 06/30 Flush 07/30 Write block 559 len 24576 data: 6 5 6 6 6 7 08/30 Flush 09/30 Write block 328 len 40960 data: 7 5 5 6 8 9 9 10 9 10 10/30 Write block 517 len 36864 data: 11 9 8 8 8 6 5 6 8 11/30 Write block 243 len 24576 data: 13 13 13 12 10 10 12/30 Write block 411 len 8192 data: 10 9 13/30 Read block 341 len 24576 14/30 Write block 131 len 28672 data: 13 13 13 12 10 13 12 15/30 Read block 34 len 12288 16/30 Read block 0 len 40960 17/30 Write block 242 len 40960 data: 12 14 14 14 13 11 11 10 10 8 18/30 Write block 540 len 16384 data: 10 10 13 9 19/30 Read block 165 len 8192 20/30 Write block 541 len 36864 data: 11 14 10 9 10 8 9 10 8 21/30 Flush 22/30 Write block 141 len 36864 data: 10 11 9 10 8 5 5 5 5 23/30 Write block 220 len 4096 data: 10 24/30 Write block 414 len 16384 data: 4 6 8 9 25/30 Write block 462 len 16384 data: 5 5 7 8 26/30 Read block 261 len 8192 27/30 Write block 524 len 40960 data: 7 9 8 8 9 7 6 10 11 11 28/30 Write block 32 len 32768 data: 11 12 14 17 18 18 20 17 29/30 Write block 274 len 8192 data: 10 10 30/30 Write block 535 len 36864 data: 12 12 10 11 10 11 12 15 11 ---------------------------------------------------------------- Crucible gen:104 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Write 6 Done Done Sent false 1003 Acked Read 1 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 10 Done Done Sent false 1009 Acked Write 9 Done Done Sent false 1010 Acked Write 6 Done Done Sent false 1011 Acked Write 2 Done Done Sent false 1012 Acked Read 6 Done Done Sent false 1013 Acked Write 7 Done Done Sent false 1014 Acked Read 3 Done Done Sent false 1015 Acked Read 10 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Write 4 Done Done Sent false 1018 Acked Read 2 Done Done Sent false 1019 Acked Write 9 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Write 1 Done Done Sent false 1023 Acked Write 4 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Read 2 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Write 8 Sent Sent Sent false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 28 34 Done 27 27 2 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:30.925423293Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1619} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 176 176 176 F F F 1 020-039 102 102 102 177 177 177 T T F <--- 2 040-059 102 102 102 177 177 177 F F F 3 060-079 103 103 103 178 178 178 F F F 4 080-099 103 103 103 178 178 178 F F F 5 100-119 100 100 100 174 174 174 F F F 6 120-139 104 104 99 182 182 173 F F F <--- 7 140-159 102 102 102 177 177 177 T T F <--- 8 160-179 92 92 92 163 163 163 F F F 9 180-199 102 102 102 176 176 176 F F F 10 200-219 102 102 102 177 177 177 F F F 11 220-239 103 103 103 178 178 178 T T F <--- 12 240-259 104 104 101 182 182 175 F F F <--- 13 260-279 96 96 96 169 169 169 T T F <--- 14 280-299 97 97 97 170 170 170 F F F 15 300-319 100 100 100 174 174 174 F F F 16 320-339 104 104 100 182 182 174 F F F <--- 17 340-359 96 96 96 169 169 169 F F F 18 360-379 104 104 94 179 179 167 F F T <--- 19 380-399 104 104 103 179 179 178 F F F <--- 20 400-419 104 104 103 182 182 178 T T F <--- 21 420-439 101 101 101 175 175 175 F F F 22 440-459 102 102 102 176 176 176 F F F 23 460-479 101 101 101 175 175 175 T T F <--- 24 480-499 96 96 96 169 169 169 F F F 25 500-519 104 104 98 182 182 172 F F F <--- 26 520-539 104 104 101 182 182 175 T T F <--- 27 540-559 104 104 94 182 182 167 T T F <--- 28 560-579 104 104 96 181 181 169 F F F <--- 29 580-599 96 96 96 169 169 169 F F F Max gen: 104, Max flush: 182 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 52 Verifying data now No change after verify Loop: 52 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 176 176 176 F F F 1 020-039 105 105 105 183 183 183 F F F 2 040-059 102 102 102 177 177 177 F F F 3 060-079 103 103 103 178 178 178 F F F 4 080-099 103 103 103 178 178 178 F F F 5 100-119 100 100 100 174 174 174 F F F 6 120-139 104 104 104 182 182 182 F F F 7 140-159 105 105 105 183 183 183 F F F 8 160-179 92 92 92 163 163 163 F F F 9 180-199 102 102 102 176 176 176 F F F 10 200-219 102 102 102 177 177 177 F F F 11 220-239 105 105 105 183 183 183 F F F 12 240-259 104 104 104 182 182 182 F F F 13 260-279 105 105 105 183 183 183 F F F 14 280-299 97 97 97 170 170 170 F F F 15 300-319 100 100 100 174 174 174 F F F 16 320-339 104 104 104 182 182 182 F F F 17 340-359 96 96 96 169 169 169 F F F 18 360-379 104 104 104 179 179 179 F F F 19 380-399 104 104 104 179 179 179 F F F 20 400-419 105 105 105 183 183 183 F F F 21 420-439 101 101 101 175 175 175 F F F 22 440-459 102 102 102 176 176 176 F F F 23 460-479 105 105 105 183 183 183 F F F 24 480-499 96 96 96 169 169 169 F F F 25 500-519 104 104 104 182 182 182 F F F 26 520-539 105 105 105 183 183 183 F F F 27 540-559 105 105 105 183 183 183 F F F 28 560-579 104 104 104 181 181 181 F F F 29 580-599 96 96 96 169 169 169 F F F Max gen: 105, Max flush: 183 Begin loop 53 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.344017743Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.390197505Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.390832005Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.390891898Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.390913855Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.390941572Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627} {"msg":"Crucible 86085255-b056-46a3-9ed4-8a47eb1e1b0b has session id: d2b6e442-0b73-4b06-b7d8-11a577846fe5","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.391478744Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"Upstairs opts: Upstairs UUID: 86085255-b056-46a3-9ed4-8a47eb1e1b0b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.391528578Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"Crucible stats registered with UUID: 86085255-b056-46a3-9ed4-8a47eb1e1b0b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.391548025Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:35.391564103Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.391802293Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"io task","client":"0","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.391852307Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"io task","client":"1","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.391889252Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"io task","client":"2","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.392201303Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"task":"crutest"} {"msg":"The guest has requested activation with gen:106","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.392229789Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627} {"msg":"86085255-b056-46a3-9ed4-8a47eb1e1b0b active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.392289972Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.392310289Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"client":"0","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.392336556Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"client":"1","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.392367942Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"client":"2","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.392401748Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"io task","client":"2","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.392437573Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"io task","client":"0","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.392473388Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"io task","client":"1","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"ds_connection connected from Ok(127.0.0.1:49734)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.392520293Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"io task","client":"0","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"ds_connection connected from Ok(127.0.0.1:62183)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.392554218Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"io task","client":"2","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"ds_connection connected from Ok(127.0.0.1:52042)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.392668634Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"io task","client":"1","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.393716412Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"client":"2","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.393818649Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"client":"1","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.393852585Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"client":"0","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394281551Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394374479Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"[0]R flush_numbers[0..12]: [176, 183, 177, 178, 178, 174, 182, 183, 163, 176, 177, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394439381Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"downstairs","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"[0]R generation[0..12]: [102, 105, 102, 103, 103, 100, 104, 105, 92, 102, 102, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394462778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"downstairs","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394483005Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"downstairs","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"[1]R flush_numbers[0..12]: [176, 183, 177, 178, 178, 174, 182, 183, 163, 176, 177, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394501953Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"downstairs","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"[1]R generation[0..12]: [102, 105, 102, 103, 103, 100, 104, 105, 92, 102, 102, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.39452075Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"downstairs","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394545638Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"downstairs","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"[2]R flush_numbers[0..12]: [176, 183, 177, 178, 178, 174, 182, 183, 163, 176, 177, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394564985Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"downstairs","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"[2]R generation[0..12]: [102, 105, 102, 103, 103, 100, 104, 105, 92, 102, 102, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394585533Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"downstairs","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394615569Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"downstairs","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"Max found gen is 106","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394633326Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"downstairs","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {Wait for a query_work_queue command to finish before sending IO "msg":"Generation requested: 106 >= found:106","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394656764Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"downstairs","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"Next flush: 184","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394675981Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"downstairs","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394692259Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"":"downstairs","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394709237Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394724605Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"client":"0","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394741393Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"client":"1","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394758281Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"client":"2","":"downstairs","session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394774978Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394790967Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"86085255-b056-46a3-9ed4-8a47eb1e1b0b is now active with session: d2b6e442-0b73-4b06-b7d8-11a577846fe5","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394809194Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394824342Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627,"session_id":"d2b6e442-0b73-4b06-b7d8-11a577846fe5"} {"msg":"The guest has finished waiting for activation with:106","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.39484022Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627} {"msg":"Activated sub_volume 86085255-b056-46a3-9ed4-8a47eb1e1b0b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:35.394854188Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 187 len 28672 data: 11 12 8 6 7 8 7 02/30 Write block 390 len 40960 data: 8 9 10 9 8 7 6 7 6 7 03/30 Flush 04/30 Write block 363 len 4096 data: 6 05/30 Write block 118 len 36864 data: 13 13 14 8 8 10 9 9 8 06/30 Read block 241 len 4096 07/30 Read block 147 len 32768 08/30 Write block 407 len 32768 data: 7 8 9 10 11 10 8 5 09/30 Read block 401 len 20480 10/30 Read block 332 len 32768 11/30 Read block 566 len 4096 12/30 Write block 104 len 24576 data: 8 8 7 8 7 7 13/30 Read block 106 len 12288 14/30 Write block 121 len 4096 data: 9 15/30 Flush 16/30 Write block 160 len 20480 data: 8 8 9 7 8 17/30 Write block 497 len 12288 data: 4 6 8 18/30 Write block 7 len 36864 data: 5 6 6 6 8 7 6 5 3 19/30 Read block 487 len 16384 20/30 Read block 135 len 16384 21/30 Read block 563 len 32768 22/30 Read block 175 len 36864 23/30 Read block 15 len 32768 24/30 Read block 286 len 12288 25/30 Read block 483 len 32768 26/30 Read block 301 len 16384 27/30 Write block 289 len 28672 data: 8 9 8 8 9 8 7 28/30 Read block 217 len 24576 29/30 Read block 260 len 8192 30/30 Write block 459 len 36864 data: 9 5 5 6 6 8 9 8 10 ---------------------------------------------------------------- Crucible gen:106 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Sent Done false 1001 Acked Write 10 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 1 Done Sent Done false 1004 Acked Write 9 Done Sent Done false 1005 Acked Read 1 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Write 8 Done Sent Done false 1008 Acked Read 5 Done Sent Done false 1009 Acked Read 8 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Read 3 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Read 3 Done Sent Done false 1024 Acked Read 8 Done Sent Done false 1025 Acked Read 4 Done Sent Done false 1026 Acked Write 7 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 1 32 Done 29 0 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) None Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:35.414748112Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1627} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 176 176 176 T F T <--- 1 020-039 105 105 105 183 183 183 F F F 2 040-059 102 102 102 177 177 177 F F F 3 060-079 103 103 103 178 178 178 F F F 4 080-099 103 103 103 178 178 178 F F F 5 100-119 106 100 106 185 174 185 F F F <--- 6 120-139 106 104 106 185 182 185 F F F <--- 7 140-159 105 105 105 183 183 183 F F F 8 160-179 92 92 92 163 163 163 T F T <--- 9 180-199 106 102 106 184 176 184 F F F <--- 10 200-219 102 102 102 177 177 177 F F F 11 220-239 105 105 105 183 183 183 F F F 12 240-259 104 104 104 182 182 182 F F F 13 260-279 105 105 105 183 183 183 F F F 14 280-299 97 97 97 170 170 170 T F T <--- 15 300-319 100 100 100 174 174 174 F F F 16 320-339 104 104 104 182 182 182 F F F 17 340-359 96 96 96 169 169 169 F F F 18 360-379 106 104 106 185 179 185 F F F <--- 19 380-399 106 104 106 184 179 184 F F F <--- 20 400-419 106 105 106 185 183 185 F F F <--- 21 420-439 101 101 101 175 175 175 F F F 22 440-459 102 102 102 176 176 176 T F T <--- 23 460-479 105 105 105 183 183 183 T F T <--- 24 480-499 96 96 96 169 169 169 T F T <--- 25 500-519 104 104 104 182 182 182 F F F 26 520-539 105 105 105 183 183 183 F F F 27 540-559 105 105 105 183 183 183 F F F 28 560-579 104 104 104 181 181 181 F F F 29 580-599 96 96 96 169 169 169 F F F Max gen: 106, Max flush: 185 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 53 Verifying data now No change after verify Loop: 53 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 107 107 107 186 186 186 F F F 1 020-039 105 105 105 183 183 183 F F F 2 040-059 102 102 102 177 177 177 F F F 3 060-079 103 103 103 178 178 178 F F F 4 080-099 103 103 103 178 178 178 F F F 5 100-119 106 106 106 185 185 185 F F F 6 120-139 106 106 106 185 185 185 F F F 7 140-159 105 105 105 183 183 183 F F F 8 160-179 107 107 107 186 186 186 F F F 9 180-199 106 106 106 184 184 184 F F F 10 200-219 102 102 102 177 177 177 F F F 11 220-239 105 105 105 183 183 183 F F F 12 240-259 104 104 104 182 182 182 F F F 13 260-279 105 105 105 183 183 183 F F F 14 280-299 107 107 107 186 186 186 F F F 15 300-319 100 100 100 174 174 174 F F F 16 320-339 104 104 104 182 182 182 F F F 17 340-359 96 96 96 169 169 169 F F F 18 360-379 106 106 106 185 185 185 F F F 19 380-399 106 106 106 184 184 184 F F F 20 400-419 106 106 106 185 185 185 F F F 21 420-439 101 101 101 175 175 175 F F F 22 440-459 107 107 107 186 186 186 F F F 23 460-479 107 107 107 186 186 186 F F F 24 480-499 107 107 107 186 186 186 F F F 25 500-519 104 104 104 182 182 182 F F F 26 520-539 105 105 105 183 183 183 F F F 27 540-559 105 105 105 183 183 183 F F F 28 560-579 104 104 104 181 181 181 F F F 29 580-599 96 96 96 169 169 169 F F F Max gen: 107, Max flush: 186 Begin loop 54 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.838967394Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.886281063Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.886996003Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.887054335Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.887078592Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.887106729Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635} {"msg":"Crucible 4ac91dd4-3289-41a4-b828-2578ff6816b8 has session id: 3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.887735979Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"Upstairs opts: Upstairs UUID: 4ac91dd4-3289-41a4-b828-2578ff6816b8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.887784633Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"Crucible stats registered with UUID: 4ac91dd4-3289-41a4-b828-2578ff6816b8","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.887803431Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:39.887819539Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888041881Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"io task","client":"1","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888088975Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"io task","client":"2","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888121731Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"io task","client":"0","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888386807Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"task":"crutest"} {"msg":"The guest has requested activation with gen:108","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888420673Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635} {"msg":"4ac91dd4-3289-41a4-b828-2578ff6816b8 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888452389Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888468877Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"client":"0","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888486215Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"client":"1","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888503183Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"client":"2","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888519651Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"io task","client":"2","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888594441Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"io task","client":"0","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888705937Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"io task","client":"1","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"ds_connection connected from Ok(127.0.0.1:37051)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888751621Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"io task","client":"2","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"ds_connection connected from Ok(127.0.0.1:63285)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888787997Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"io task","client":"1","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"ds_connection connected from Ok(127.0.0.1:58066)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.888824612Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"io task","client":"0","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.889976057Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"client":"0","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.890009163Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"client":"1","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.890549615Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.890581261Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"client":"2","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.890599498Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"[0]R flush_numbers[0..12]: [186, 183, 177, 178, 178, 185, 185, 183, 186, 184, 177, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891152299Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"downstairs","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"[0]R generation[0..12]: [107, 105, 102, 103, 103, 106, 106, 105, 107, 106, 102, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891180555Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"downstairs","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891200823Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"downstairs","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"[1]R flush_numbers[0..12]: [186, 183, 177, 178, 178, 185, 185, 183, 186, 184, 177, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.89122123Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"downstairs","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"[1]R generation[0..12]: [107, 105, 102, 103, 103, 106, 106, 105, 107, 106, 102, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891249747Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"downstairs","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891270314Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"downstairs","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"[2]R flush_numbers[0..12]: [186, 183, 177, 178, 178, 185, 185, 183, 186, 184, 177, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891289671Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"downstairs","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"[2]R generation[0..12]: [107, 105, 102, 103, 103, 106, 106, 105, 107, 106, 102, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891308409Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"downstairs","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891327787Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"downstairs","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"Max found gen is 108","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-11-06T20:15:39.891344135Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"downstairs","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"Generation requested: 108 >= found:108","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891367852Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"downstairs","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"Next flush: 187","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.89138398Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"downstairs","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891400178Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"":"downstairs","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891417295Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891432584Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"client":"0","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891449662Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"client":"1","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891466739Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"client":"2","":"downstairs","session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891483777Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891502955Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"4ac91dd4-3289-41a4-b828-2578ff6816b8 is now active with session: 3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891524192Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.89154001Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635,"session_id":"3a1e2a9c-7d1d-4bbe-9d84-c7fdfd3b648f"} {"msg":"The guest has finished waiting for activation with:108","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891556588Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635} {"msg":"Activated sub_volume 4ac91dd4-3289-41a4-b828-2578ff6816b8","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:39.891570696Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 502 len 20480 data: 10 10 8 10 10 02/30 Flush 03/30 Write block 150 len 40960 data: 4 4 5 5 6 7 5 5 5 8 04/30 Read block 266 len 4096 05/30 Read block 39 len 16384 06/30 Write block 549 len 20480 data: 9 9 10 9 10 07/30 Read block 452 len 32768 08/30 Read block 344 len 24576 09/30 Read block 491 len 12288 10/30 Write block 264 len 4096 data: 3 11/30 Flush 12/30 Write block 23 len 24576 data: 11 11 12 11 10 10 13/30 Read block 483 len 36864 14/30 Read block 35 len 4096 15/30 Read block 383 len 8192 16/30 Read block 434 len 8192 17/30 Write block 592 len 20480 data: 10 8 8 9 10 18/30 Write block 87 len 20480 data: 7 8 10 9 8 19/30 Read block 439 len 20480 20/30 Read block 460 len 40960 21/30 Flush 22/30 Flush 23/30 Write block 539 len 8192 data: 11 12 24/30 Read block 349 len 16384 25/30 Write block 206 len 8192 data: 10 8 26/30 Read block 80 len 4096 27/30 Write block 504 len 20480 data: 9 11 11 7 9 28/30 Write block 266 len 16384 data: 9 11 10 10 29/30 Read block 575 len 36864 30/30 Read block 563 len 4096 ---------------------------------------------------------------- Crucible gen:108 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Sent false 1001 Acked Flush 0 Done Done Sent false 1002 Acked Write 10 Done Done Sent false 1003 Acked Read 1 Done Done Sent false 1004 Acked Read 4 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Read 8 Done Done Sent false 1007 Acked Read 6 Done Done Sent false 1008 Acked Read 3 Done Done Sent false 1009 Acked Write 1 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Read 9 Done Done Sent false 1013 Acked Read 1 Done Done Sent false 1014 Acked Read 2 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Read 10 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 2 Done Done Sent false 1023 Acked Read 4 Done Done Sent false 1024 Acked Write 2 Done Done Sent false 1025 Acked Read 1 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Read 1 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1021)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:39.910167413Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1635} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 107 107 107 186 186 186 F F F 1 020-039 108 108 105 189 189 183 F F F <--- 2 040-059 102 102 102 177 177 177 F F F 3 060-079 103 103 103 178 178 178 F F F 4 080-099 108 108 103 189 189 178 F F F <--- 5 100-119 106 106 106 185 185 185 F F F 6 120-139 106 106 106 185 185 185 F F F 7 140-159 108 108 105 188 188 183 F F F <--- 8 160-179 107 107 107 186 186 186 F F F 9 180-199 106 106 106 184 184 184 F F F 10 200-219 102 102 102 177 177 177 T T F <--- 11 220-239 105 105 105 183 183 183 F F F 12 240-259 104 104 104 182 182 182 F F F 13 260-279 108 108 105 188 188 183 T T F <--- 14 280-299 107 107 107 186 186 186 F F F 15 300-319 100 100 100 174 174 174 F F F 16 320-339 104 104 104 182 182 182 F F F 17 340-359 96 96 96 169 169 169 F F F 18 360-379 106 106 106 185 185 185 F F F 19 380-399 106 106 106 184 184 184 F F F 20 400-419 106 106 106 185 185 185 F F F 21 420-439 101 101 101 175 175 175 F F F 22 440-459 107 107 107 186 186 186 F F F 23 460-479 107 107 107 186 186 186 F F F 24 480-499 107 107 107 186 186 186 F F F 25 500-519 108 108 104 187 187 182 T T F <--- 26 520-539 105 105 105 183 183 183 T T F <--- 27 540-559 108 108 105 188 188 183 T T F <--- 28 560-579 104 104 104 181 181 181 F F F 29 580-599 108 108 96 189 189 169 F F F <--- Max gen: 108, Max flush: 189 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 54 Verifying data now No change after verify Loop: 54 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 107 107 107 186 186 186 F F F 1 020-039 108 108 108 189 189 189 F F F 2 040-059 102 102 102 177 177 177 F F F 3 060-079 103 103 103 178 178 178 F F F 4 080-099 108 108 108 189 189 189 F F F 5 100-119 106 106 106 185 185 185 F F F 6 120-139 106 106 106 185 185 185 F F F 7 140-159 108 108 108 188 188 188 F F F 8 160-179 107 107 107 186 186 186 F F F 9 180-199 106 106 106 184 184 184 F F F 10 200-219 109 109 109 190 190 190 F F F 11 220-239 105 105 105 183 183 183 F F F 12 240-259 104 104 104 182 182 182 F F F 13 260-279 109 109 109 190 190 190 F F F 14 280-299 107 107 107 186 186 186 F F F 15 300-319 100 100 100 174 174 174 F F F 16 320-339 104 104 104 182 182 182 F F F 17 340-359 96 96 96 169 169 169 F F F 18 360-379 106 106 106 185 185 185 F F F 19 380-399 106 106 106 184 184 184 F F F 20 400-419 106 106 106 185 185 185 F F F 21 420-439 101 101 101 175 175 175 F F F 22 440-459 107 107 107 186 186 186 F F F 23 460-479 107 107 107 186 186 186 F F F 24 480-499 107 107 107 186 186 186 F F F 25 500-519 109 109 109 190 190 190 F F F 26 520-539 109 109 109 190 190 190 F F F 27 540-559 109 109 109 190 190 190 F F F 28 560-579 104 104 104 181 181 181 F F F 29 580-599 108 108 108 189 189 189 F F F Max gen: 109, Max flush: 190 Begin loop 55 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.298565527Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.344156273Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.344784994Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.344845846Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.344867684Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.34489567Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643} {"msg":"Crucible 7f1a1ff5-ad67-4a00-8b84-973b9c1da975 has session id: 08b47cc8-2ce8-47a7-8830-95105d2e2cca","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.345419064Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"Upstairs opts: Upstairs UUID: 7f1a1ff5-ad67-4a00-8b84-973b9c1da975, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.345475797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"Crucible stats registered with UUID: 7f1a1ff5-ad67-4a00-8b84-973b9c1da975","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.345506003Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:44.3455322Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.345724976Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"io task","client":"0","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.345781789Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"io task","client":"1","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.345807275Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"io task","client":"2","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.345910912Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"task":"crutest"} {"msg":"The guest has requested activation with gen:110","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.34593247Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643} {"msg":"7f1a1ff5-ad67-4a00-8b84-973b9c1da975 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.345993042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.346023718Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"client":"0","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.346043485Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"client":"1","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.346061193Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"client":"2","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.346077791Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"io task","client":"2","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.346103078Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"io task","client":"0","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.346130734Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"io task","client":"1","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"ds_connection connected from Ok(127.0.0.1:45994)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.346302393Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"io task","client":"2","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"ds_connection connected from Ok(127.0.0.1:37415)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.346349837Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"io task","client":"1","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"ds_connection connected from Ok(127.0.0.1:48091)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.346385872Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"io task","client":"0","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.347629386Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"client":"0","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.347661992Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"client":"2","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348145001Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"client":"1","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348341526Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348372692Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"[0]R flush_numbers[0..12]: [186, 189, 177, 178, 189, 185, 185, 188, 186, 184, 190, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348646657Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"downstairs","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"[0]R generation[0..12]: [107, 108, 102, 103, 108, 106, 106, 108, 107, 106, 109, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348671574Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"downstairs","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348691862Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"downstairs","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"[1]R flush_numbers[0..12]: [186, 189, 177, 178, 189, 185, 185, 188, 186, 184, 190, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348711369Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"downstairs","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"[1]R generation[0..12]: [107, 108, 102, 103, 108, 106, 106, 108, 107, 106, 109, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348730817Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"downstairs","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348750624Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"downstairs","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"[2]R flush_numbers[0..12]: [186, 189, 177, 178, 189, 185, 185, 188, 186, 184, 190, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348769722Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"downstairs","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"[2]R generation[0..12]: [107, 108, 102, 103, 108, 106, 106, 108, 107, 106, 109, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348788639Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"downstairs","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348808247Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"downstairs","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"Max found gen is 110","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348824545Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"downstairs","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"Generation requested: 110 >= found:110","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348841543Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"downstairs","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} Wait for a query_work_queue command to finish before sending IO {"msg":"Next flush: 191","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348869309Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"downstairs","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348886227Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"":"downstairs","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348903115Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348918233Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"client":"0","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348934891Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"client":"1","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348951359Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"client":"2","":"downstairs","session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348968327Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.348984285Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"7f1a1ff5-ad67-4a00-8b84-973b9c1da975 is now active with session: 08b47cc8-2ce8-47a7-8830-95105d2e2cca","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.349018351Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.349039418Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643,"session_id":"08b47cc8-2ce8-47a7-8830-95105d2e2cca"} {"msg":"The guest has finished waiting for activation with:110","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.349056046Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643} {"msg":"Activated sub_volume 7f1a1ff5-ad67-4a00-8b84-973b9c1da975","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:44.349070034Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 347 len 16384 data: 5 6 6 8 03/30 Read block 135 len 20480 04/30 Write block 4 len 36864 data: 3 3 5 6 7 7 7 9 8 05/30 Read block 201 len 40960 06/30 Read block 452 len 20480 07/30 Read block 419 len 12288 08/30 Write block 177 len 28672 data: 9 11 15 17 18 18 18 09/30 Read block 321 len 8192 10/30 Read block 31 len 36864 11/30 Write block 251 len 12288 data: 9 7 8 12/30 Flush 13/30 Write block 451 len 36864 data: 14 10 10 12 13 13 11 10 10 14/30 Write block 488 len 8192 data: 6 5 15/30 Read block 444 len 28672 16/30 Flush 17/30 Write block 194 len 40960 data: 8 7 9 13 12 12 12 13 10 10 18/30 Read block 117 len 20480 19/30 Read block 502 len 12288 20/30 Read block 516 len 36864 21/30 Write block 44 len 12288 data: 12 13 12 22/30 Flush 23/30 Write block 470 len 28672 data: 9 10 12 12 10 11 9 24/30 Read block 262 len 8192 25/30 Read block 105 len 20480 26/30 Write block 114 len 40960 data: 13 11 18 15 14 14 15 10 9 11 27/30 Read block 395 len 28672 28/30 Read block 89 len 20480 29/30 Write block 74 len 24576 data: 4 4 6 6 9 9 30/30 Write block 40 len 4096 data: 19 ---------------------------------------------------------------- Crucible gen:110 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 4 Done Done Done false 1002 Acked Read 5 Done Sent Done false 1003 Acked Write 9 Done Sent Done false 1004 Acked Read 10 Done Sent Done false 1005 Acked Read 5 Done Sent Done false 1006 Acked Read 3 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Read 9 Done Sent Done false 1010 Acked Write 3 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 9 Done Sent Done false 1013 Acked Write 2 Done Sent Done false 1014 Acked Read 7 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 10 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Read 3 Done Sent Done false 1019 Acked Read 9 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Read 2 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Read 7 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 28 2 32 Done 27 1 27 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1000)) Some(JobId(1021)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:44.370067709Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1643} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 107 110 192 186 192 F F F <--- 1 020-039 108 108 108 189 189 189 F F F 2 040-059 110 102 110 194 177 194 T F T <--- 3 060-079 103 103 103 178 178 178 T F T <--- 4 080-099 108 108 108 189 189 189 F F F 5 100-119 106 106 106 185 185 185 T F T <--- 6 120-139 106 106 106 185 185 185 T F T <--- 7 140-159 108 108 108 188 188 188 F F F 8 160-179 110 107 110 192 186 192 F F F <--- 9 180-199 110 106 110 194 184 194 F F F <--- 10 200-219 110 109 110 194 190 194 F F F <--- 11 220-239 105 105 105 183 183 183 F F F 12 240-259 110 104 110 192 182 192 F F F <--- 13 260-279 109 109 109 190 190 190 F F F 14 280-299 107 107 107 186 186 186 F F F 15 300-319 100 100 100 174 174 174 F F F 16 320-339 104 104 104 182 182 182 F F F 17 340-359 110 96 110 192 169 192 F T F <--- 18 360-379 106 106 106 185 185 185 F F F 19 380-399 106 106 106 184 184 184 F F F 20 400-419 106 106 106 185 185 185 F F F 21 420-439 101 101 101 175 175 175 F F F 22 440-459 110 107 110 193 186 193 F F F <--- 23 460-479 107 107 107 186 186 186 T F T <--- 24 480-499 110 107 110 193 186 193 F F F <--- 25 500-519 109 109 109 190 190 190 F F F 26 520-539 109 109 109 190 190 190 F F F 27 540-559 109 109 109 190 190 190 F F F 28 560-579 104 104 104 181 181 181 F F F 29 580-599 108 108 108 189 189 189 F F F Max gen: 110, Max flush: 194 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 55 Verifying data now No change after verify Loop: 55 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 110 110 192 192 192 F F F 1 020-039 108 108 108 189 189 189 F F F 2 040-059 111 111 111 195 195 195 F F F 3 060-079 111 111 111 195 195 195 F F F 4 080-099 108 108 108 189 189 189 F F F 5 100-119 111 111 111 195 195 195 F F F 6 120-139 111 111 111 195 195 195 F F F 7 140-159 108 108 108 188 188 188 F F F 8 160-179 110 110 110 192 192 192 F F F 9 180-199 110 110 110 194 194 194 F F F 10 200-219 110 110 110 194 194 194 F F F 11 220-239 105 105 105 183 183 183 F F F 12 240-259 110 110 110 192 192 192 F F F 13 260-279 109 109 109 190 190 190 F F F 14 280-299 107 107 107 186 186 186 F F F 15 300-319 100 100 100 174 174 174 F F F 16 320-339 104 104 104 182 182 182 F F F 17 340-359 110 110 110 192 192 192 F F F 18 360-379 106 106 106 185 185 185 F F F 19 380-399 106 106 106 184 184 184 F F F 20 400-419 106 106 106 185 185 185 F F F 21 420-439 101 101 101 175 175 175 F F F 22 440-459 110 110 110 193 193 193 F F F 23 460-479 111 111 111 195 195 195 F F F 24 480-499 110 110 110 193 193 193 F F F 25 500-519 109 109 109 190 190 190 F F F 26 520-539 109 109 109 190 190 190 F F F 27 540-559 109 109 109 190 190 190 F F F 28 560-579 104 104 104 181 181 181 F F F 29 580-599 108 108 108 189 189 189 F F F Max gen: 111, Max flush: 195 Begin loop 56 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.78015794Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.825654108Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.826266561Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.826315704Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.826337112Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.826360439Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651} {"msg":"Crucible a2393cda-1eba-48d0-b827-7e23b1c2690f has session id: 2023514b-63a9-4034-a914-13c3be6651e4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.826957494Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"Upstairs opts: Upstairs UUID: a2393cda-1eba-48d0-b827-7e23b1c2690f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.827003468Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"Crucible stats registered with UUID: a2393cda-1eba-48d0-b827-7e23b1c2690f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.827023455Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:48.827040093Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.8272249Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"0","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.827270914Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"2","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.8273021Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"1","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.82754213Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"task":"crutest"} {"msg":"The guest has requested activation with gen:112","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.827570797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651} {"msg":"a2393cda-1eba-48d0-b827-7e23b1c2690f active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.827626629Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.827652476Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"client":"0","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.827671944Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"client":"1","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.827688802Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"client":"2","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.827708929Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"2","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.827727567Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"0","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.827744684Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"1","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:48.827878558Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"2","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:55966)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.827925832Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"1","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:51210)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.827975046Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"0","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:48.828013191Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"2","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:48.828052896Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.828081892Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"2","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.828112418Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"2","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.829012245Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"client":"1","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.82912521Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"client":"0","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.829573754Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:48.829673372Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.828905078Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"2","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"ds_connection connected from Ok(127.0.0.1:39559)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.829026883Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"io task","client":"2","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.830904266Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"client":"2","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"[0]R flush_numbers[0..12]: [192, 189, 195, 195, 189, 195, 195, 188, 192, 194, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831538856Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"downstairs","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"[0]R generation[0..12]: [110, 108, 111, 111, 108, 111, 111, 108, 110, 110, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831564163Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"downstairs","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.83158504Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"downstairs","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"[1]R flush_numbers[0..12]: [192, 189, 195, 195, 189, 195, 195, 188, 192, 194, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831609047Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"downstairs","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"[1]R generation[0..12]: [110, 108, 111, 111, 108, 111, 111, 108, 110, 110, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831633194Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"downstairs","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831657791Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"downstairs","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"[2]R flush_numbers[0..12]: [192, 189, 195, 195, 189, 195, 195, 188, 192, 194, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831681588Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"downstairs","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"[2]R generation[0..12]: [110, 108, 111, 111, 108, 111, 111, 108, 110, 110, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831704885Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"downstairs","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831729582Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"downstairs","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"Max found gen is 112","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831750179Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"downstairs","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"Generation requested: 112 >= found:112","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831771457Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"downstairs","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"Next flush: 196","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831795994Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"downstairs","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831816571Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"":"downstairs","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831837708Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831856666Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651Wait for a query_work_queue command to finish before sending IO ,"client":"0","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831884653Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"client":"1","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.83190562Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"client":"2","":"downstairs","session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831926967Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831946985Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"a2393cda-1eba-48d0-b827-7e23b1c2690f is now active with session: 2023514b-63a9-4034-a914-13c3be6651e4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.831969212Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.83198817Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651,"session_id":"2023514b-63a9-4034-a914-13c3be6651e4"} {"msg":"The guest has finished waiting for activation with:112","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:15:58.832007927Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651} {"msg":"Activated sub_volume a2393cda-1eba-48d0-b827-7e23b1c2690f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:15:58.832027725Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 323 len 32768 data: 7 8 9 9 8 8 6 6 02/30 Write block 552 len 36864 data: 10 11 11 10 8 8 7 7 6 03/30 Read block 234 len 36864 04/30 Write block 105 len 28672 data: 9 8 9 8 8 9 11 05/30 Read block 341 len 36864 06/30 Read block 197 len 20480 07/30 Read block 496 len 36864 08/30 Flush 09/30 Flush 10/30 Write block 166 len 36864 data: 7 7 9 9 6 6 5 4 7 11/30 Read block 197 len 4096 12/30 Read block 410 len 36864 13/30 Write block 184 len 16384 data: 13 13 13 12 14/30 Flush 15/30 Write block 275 len 16384 data: 11 9 10 10 16/30 Write block 579 len 12288 data: 8 6 6 17/30 Read block 291 len 8192 18/30 Write block 381 len 24576 data: 6 5 5 5 6 5 19/30 Write block 583 len 12288 data: 5 4 7 20/30 Read block 278 len 4096 21/30 Read block 330 len 24576 22/30 Write block 405 len 40960 data: 9 8 8 9 10 11 12 11 9 6 23/30 Write block 390 len 24576 data: 9 10 11 10 9 8 24/30 Read block 340 len 12288 25/30 Write block 457 len 40960 data: 12 11 11 6 6 7 7 9 10 9 26/30 Read block 438 len 28672 27/30 Write block 132 len 12288 data: 14 14 13 28/30 Write block 289 len 32768 data: 9 10 9 9 10 9 8 6 29/30 Read block 335 len 4096 30/30 Read block 227 len 16384 ---------------------------------------------------------------- Crucible gen:112 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Read 9 Done Done Done false 1003 Acked Write 7 Done Done Sent false 1004 Acked Read 9 Done Done Sent false 1005 Acked Read 5 Done Done Sent false 1006 Acked Read 9 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 9 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Read 9 Done Done Sent false 1012 Acked Write 4 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Write 3 Done Done Sent false 1016 Acked Read 2 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Write 3 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Write 10 Done Done Sent false 1022 Acked Write 6 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Read 7 Done Done Sent false 1026 Acked Write 3 Done Done Sent false 1027 Acked Write 8 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 27 28 Done 30 29 3 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1013)) Some(JobId(1013)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:15:58.852818865Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1651} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 110 110 192 192 192 F F F 1 020-039 108 108 108 189 189 189 F F F 2 040-059 111 111 111 195 195 195 F F F 3 060-079 111 111 111 195 195 195 F F F 4 080-099 108 108 108 189 189 189 F F F 5 100-119 112 112 111 196 196 195 F F F <--- 6 120-139 111 111 111 195 195 195 T T F <--- 7 140-159 108 108 108 188 188 188 F F F 8 160-179 112 112 110 198 198 192 F F F <--- 9 180-199 112 112 110 198 198 194 F F F <--- 10 200-219 110 110 110 194 194 194 F F F 11 220-239 105 105 105 183 183 183 F F F 12 240-259 110 110 110 192 192 192 F F F 13 260-279 109 109 109 190 190 190 T T F <--- 14 280-299 107 107 107 186 186 186 T T F <--- 15 300-319 100 100 100 174 174 174 F F F 16 320-339 112 112 104 196 196 182 F F T <--- 17 340-359 110 110 110 192 192 192 F F F 18 360-379 106 106 106 185 185 185 F F F 19 380-399 106 106 106 184 184 184 T T F <--- 20 400-419 106 106 106 185 185 185 T T F <--- 21 420-439 101 101 101 175 175 175 F F F 22 440-459 110 110 110 193 193 193 T T F <--- 23 460-479 111 111 111 195 195 195 T T F <--- 24 480-499 110 110 110 193 193 193 F F F 25 500-519 109 109 109 190 190 190 F F F 26 520-539 109 109 109 190 190 190 F F F 27 540-559 112 112 109 196 196 190 F F T <--- 28 560-579 112 112 104 196 196 181 T T T <--- 29 580-599 108 108 108 189 189 189 T T F <--- Max gen: 112, Max flush: 198 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 56 Verifying data now No change after verify Loop: 56 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 110 110 192 192 192 F F F 1 020-039 108 108 108 189 189 189 F F F 2 040-059 111 111 111 195 195 195 F F F 3 060-079 111 111 111 195 195 195 F F F 4 080-099 108 108 108 189 189 189 F F F 5 100-119 112 112 112 196 196 196 F F F 6 120-139 113 113 113 199 199 199 F F F 7 140-159 108 108 108 188 188 188 F F F 8 160-179 112 112 112 198 198 198 F F F 9 180-199 112 112 112 198 198 198 F F F 10 200-219 110 110 110 194 194 194 F F F 11 220-239 105 105 105 183 183 183 F F F 12 240-259 110 110 110 192 192 192 F F F 13 260-279 113 113 113 199 199 199 F F F 14 280-299 113 113 113 199 199 199 F F F 15 300-319 100 100 100 174 174 174 F F F 16 320-339 112 112 112 196 196 196 F F F 17 340-359 110 110 110 192 192 192 F F F 18 360-379 106 106 106 185 185 185 F F F 19 380-399 113 113 113 199 199 199 F F F 20 400-419 113 113 113 199 199 199 F F F 21 420-439 101 101 101 175 175 175 F F F 22 440-459 113 113 113 199 199 199 F F F 23 460-479 113 113 113 199 199 199 F F F 24 480-499 110 110 110 193 193 193 F F F 25 500-519 109 109 109 190 190 190 F F F 26 520-539 109 109 109 190 190 190 F F F 27 540-559 112 112 112 196 196 196 F F F 28 560-579 113 113 113 199 199 199 F F F 29 580-599 113 113 113 199 199 199 F F F Max gen: 113, Max flush: 199 Begin loop 57 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.270826979Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:03.314811708Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.315194859Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.346951178Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.34749266Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.347560452Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.347580569Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.347608296Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659} {"msg":"Crucible 128b6ee0-dd3d-43d7-964d-d1ada1b8e45d has session id: 113ecfed-7f47-4aea-bdeb-b478c8e6ae3e","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.348171195Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"Upstairs opts: Upstairs UUID: 128b6ee0-dd3d-43d7-964d-d1ada1b8e45d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.348224808Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"Crucible stats registered with UUID: 128b6ee0-dd3d-43d7-964d-d1ada1b8e45d","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.348249975Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:03.348270942Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.348506243Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"io task","client":"2","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.348562726Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"io task","client":"1","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.348586703Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"io task","client":"0","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.34868629Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"task":"crutest"} {"msg":"The guest has requested activation with gen:114","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.348724135Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659} {"msg":"128b6ee0-dd3d-43d7-964d-d1ada1b8e45d active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.348789437Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.348819483Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"client":"0","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.34884321Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"client":"1","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.348861448Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"client":"2","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.348878596Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"io task","client":"2","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.348896133Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"io task","client":"0","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.348914771Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"io task","client":"1","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"ds_connection connected from Ok(127.0.0.1:42884)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.349180118Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"io task","client":"0","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"ds_connection connected from Ok(127.0.0.1:60927)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.349230371Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"io task","client":"1","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"ds_connection connected from Ok(127.0.0.1:42843)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.349267497Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"io task","client":"2","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.35027667Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"client":"2","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.350311465Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"client":"1","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.350427401Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"client":"0","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.350882033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.350977771Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"[0]R flush_numbers[0..12]: [192, 189, 195, 195, 189, 196, 199, 188, 198, 198, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351116944Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"downstairs","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"[0]R generation[0..12]: [110, 108, 111, 111, 108, 112, 113, 108, 112, 112, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.35114695Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"downstairs","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351171257Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"downstairs","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"[1]R flush_numbers[0..12]: [192, 189, 195, 195, 189, 196, 199, 188, 198, 198, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351191014Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"downstairs","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"[1]R generation[0..12]: [110, 108, 111, 111, 108, 112, 113, 108, 112, 112, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351210012Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"downstairs","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.35122971Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"downstairs","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"[2]R flush_numbers[0..12]: [192, 189, 195, 195, 189, 196, 199, 188, 198, 198, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351248707Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"downstairs","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"[2]R generation[0..12]: [110, 108, 111, 111, 108, 112, 113, 108, 112, 112, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351267415Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"downstairs","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351293682Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"Max found gen is 114","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351327527Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"downstairs","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"Generation requested: 114 >= found:114","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351346955Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"downstairs","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"Next flush: 200","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351365662Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"downstairs","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351393399Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"":"downstairs","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351412996Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351428325Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"client":"0","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351446262Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"client":"1","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351477538Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"client":"2","":"downstairs","session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351497786Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351514504Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"128b6ee0-dd3d-43d7-964d-d1ada1b8e45d is now active with session: 113ecfed-7f47-4aea-bdeb-b478c8e6ae3e","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.35154347Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351565917Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659,"session_id":"113ecfed-7f47-4aea-bdeb-b478c8e6ae3e"} {"msg":"The guest has finished waiting for activation with:114","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351582405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659} {"msg":"Activated sub_volume 128b6ee0-dd3d-43d7-964d-d1ada1b8e45d","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:03.351596073Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 512 len 4096 data: 9 02/30 Read block 227 len 36864 03/30 Write block 571 len 36864 data: 4 3 4 4 5 7 8 7 9 04/30 Flush 05/30 Write block 289 len 32768 data: 10 11 10 10 11 10 9 7 06/30 Read block 86 len 16384 07/30 Write block 470 len 32768 data: 10 11 13 13 11 12 10 8 08/30 Read block 451 len 20480 09/30 Write block 435 len 12288 data: 7 12 12 10/30 Write block 333 len 24576 data: 10 10 11 10 11 11 11/30 Write block 371 len 32768 data: 5 5 4 5 4 4 5 5 12/30 Read block 475 len 8192 13/30 Write block 264 len 16384 data: 4 4 10 12 14/30 Read block 417 len 40960 15/30 Read block 440 len 8192 16/30 Read block 386 len 20480 17/30 Write block 419 len 24576 data: 13 12 11 11 9 11 18/30 Write block 130 len 4096 data: 14 19/30 Write block 258 len 40960 data: 11 5 5 5 5 4 5 5 11 13 20/30 Read block 153 len 8192 21/30 Read block 352 len 24576 22/30 Read block 485 len 8192 23/30 Write block 119 len 8192 data: 15 16 24/30 Write block 289 len 8192 data: 11 12 25/30 Write block 91 len 36864 data: 9 7 6 7 8 8 9 7 8 26/30 Write block 575 len 8192 data: 6 8 27/30 Write block 341 len 40960 data: 6 5 5 5 5 6 6 7 7 9 28/30 Write block 497 len 12288 data: 5 7 9 29/30 Read block 498 len 28672 30/30 Write block 430 len 28672 data: 9 7 6 6 9 8 13 ---------------------------------------------------------------- Crucible gen:114 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Write 9 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 8 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Write 8 Sent Done Done false 1007 Acked Read 5 Sent Done Done false 1008 Acked Write 3 Sent Done Done false 1009 Acked Write 6 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Write 4 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Read 5 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Write 2 Sent Done Done false 1023 Acked Write 2 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Write 3 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 1 30 Done 2 29 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:03.369760905Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1659} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 110 110 192 192 192 F F F 1 020-039 108 108 108 189 189 189 F F F 2 040-059 111 111 111 195 195 195 F F F 3 060-079 111 111 111 195 195 195 F F F 4 080-099 108 108 108 189 189 189 F T T <--- 5 100-119 112 112 112 196 196 196 F T T <--- 6 120-139 113 113 113 199 199 199 F T T <--- 7 140-159 108 108 108 188 188 188 F F F 8 160-179 112 112 112 198 198 198 F F F 9 180-199 112 112 112 198 198 198 F F F 10 200-219 110 110 110 194 194 194 F F F 11 220-239 105 105 105 183 183 183 F F F 12 240-259 110 110 110 192 192 192 F T T <--- 13 260-279 113 113 113 199 199 199 F T T <--- 14 280-299 113 113 113 199 199 199 F T T <--- 15 300-319 100 100 100 174 174 174 F F F 16 320-339 112 112 112 196 196 196 F T T <--- 17 340-359 110 110 110 192 192 192 F T T <--- 18 360-379 106 106 106 185 185 185 F T T <--- 19 380-399 113 113 113 199 199 199 F F F 20 400-419 113 113 113 199 199 199 F T T <--- 21 420-439 101 101 101 175 175 175 F T T <--- 22 440-459 113 113 113 199 199 199 F F F 23 460-479 113 113 113 199 199 199 F T T <--- 24 480-499 110 110 110 193 193 193 F T T <--- 25 500-519 109 114 114 190 200 200 T F F <--- 26 520-539 109 109 109 190 190 190 F F F 27 540-559 112 112 112 196 196 196 F F F 28 560-579 113 114 114 199 200 200 F T T <--- 29 580-599 113 113 113 199 199 199 F F F Max gen: 114, Max flush: 200 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 57 Verifying data now No change after verify Loop: 57 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 110 110 192 192 192 F F F 1 020-039 108 108 108 189 189 189 F F F 2 040-059 111 111 111 195 195 195 F F F 3 060-079 111 111 111 195 195 195 F F F 4 080-099 115 115 115 201 201 201 F F F 5 100-119 115 115 115 201 201 201 F F F 6 120-139 115 115 115 201 201 201 F F F 7 140-159 108 108 108 188 188 188 F F F 8 160-179 112 112 112 198 198 198 F F F 9 180-199 112 112 112 198 198 198 F F F 10 200-219 110 110 110 194 194 194 F F F 11 220-239 105 105 105 183 183 183 F F F 12 240-259 115 115 115 201 201 201 F F F 13 260-279 115 115 115 201 201 201 F F F 14 280-299 115 115 115 201 201 201 F F F 15 300-319 100 100 100 174 174 174 F F F 16 320-339 115 115 115 201 201 201 F F F 17 340-359 115 115 115 201 201 201 F F F 18 360-379 115 115 115 201 201 201 F F F 19 380-399 113 113 113 199 199 199 F F F 20 400-419 115 115 115 201 201 201 F F F 21 420-439 115 115 115 201 201 201 F F F 22 440-459 113 113 113 199 199 199 F F F 23 460-479 115 115 115 201 201 201 F F F 24 480-499 115 115 115 201 201 201 F F F 25 500-519 114 114 114 200 200 200 F F F 26 520-539 109 109 109 190 190 190 F F F 27 540-559 112 112 112 196 196 196 F F F 28 560-579 115 115 115 201 201 201 F F F 29 580-599 113 113 113 199 199 199 F F F Max gen: 115, Max flush: 201 Begin loop 58 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.835680442Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.881036637Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.881938344Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.881996686Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.882016644Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.88204424Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667} {"msg":"Crucible 01294bf3-2328-42dc-a240-02e533682448 has session id: 957f1050-baf7-4ed6-91d9-8536f23b7558","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.882572514Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"Upstairs opts: Upstairs UUID: 01294bf3-2328-42dc-a240-02e533682448, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.882623987Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"Crucible stats registered with UUID: 01294bf3-2328-42dc-a240-02e533682448","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.882648784Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:07.882666492Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.882892003Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"io task","client":"0","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.882935108Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"io task","client":"2","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.882955646Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"io task","client":"1","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.883137972Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"task":"crutest"} {"msg":"The guest has requested activation with gen:116","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.883161389Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667} {"msg":"01294bf3-2328-42dc-a240-02e533682448 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.883230241Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.883251258Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"client":"0","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.883276065Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"client":"1","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.883306811Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"client":"2","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.883337607Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"io task","client":"2","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.883372173Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"io task","client":"0","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.883403339Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"io task","client":"1","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"ds_connection connected from Ok(127.0.0.1:46664)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.883436585Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"io task","client":"2","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"ds_connection connected from Ok(127.0.0.1:59814)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.883471461Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"io task","client":"0","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"ds_connection connected from Ok(127.0.0.1:43129)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.883502706Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"io task","client":"1","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.884762798Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"client":"0","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.884799503Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"client":"2","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.885360623Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.885389999Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.885729836Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"client":"1","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"[0]R flush_numbers[0..12]: [192, 189, 195, 195, 201, 201, 201, 188, 198, 198, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886229373Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"downstairs","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"[0]R generation[0..12]: [110, 108, 111, 111, 115, 115, 115, 108, 112, 112, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886268718Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"downstairs","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886294285Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"downstairs","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"[1]R flush_numbers[0..12]: [192, 189, 195, 195, 201, 201, 201, 188, 198, 198, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886318362Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"downstairs","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"[1]R generation[0..12]: [110, 108, 111, 111, 115, 115, 115, 108, 112, 112, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886342209Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"downstairs","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886367136Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"downstairs","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"[2]R flush_numbers[0..12]: [192, 189, 195, 195, 201, 201, 201, 188, 198, 198, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886390983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"downstairs","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"[2]R generation[0..12]: [110, 108, 111, 111, 115, 115, 115, 108, 112, 112, 110, 105]","v":0,"name"Wait for a query_work_queue command to finish before sending IO :"crucible","level":30,"time":"2025-11-06T20:16:07.886425128Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"downstairs","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886450145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"downstairs","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"Max found gen is 116","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886470553Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"downstairs","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"Generation requested: 116 >= found:116","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.88649181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"downstairs","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"Next flush: 202","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886512038Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"downstairs","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886529595Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"":"downstairs","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886547103Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886562451Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"client":"0","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 "msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886583069Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"client":"1","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886599996Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"client":"2","":"downstairs","session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886617094Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886633132Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"01294bf3-2328-42dc-a240-02e533682448 is now active with session: 957f1050-baf7-4ed6-91d9-8536f23b7558","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.88665094Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886666378Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667,"session_id":"957f1050-baf7-4ed6-91d9-8536f23b7558"} {"msg":"The guest has finished waiting for activation with:116","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886682736Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667} {"msg":"Activated sub_volume 01294bf3-2328-42dc-a240-02e533682448","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:07.886696574Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 146 len 12288 data: 6 6 6 03/30 Read block 65 len 8192 04/30 Write block 70 len 36864 data: 6 6 6 3 5 5 7 7 10 05/30 Write block 145 len 4096 data: 9 06/30 Read block 108 len 28672 07/30 Read block 52 len 24576 08/30 Read block 561 len 24576 09/30 Write block 301 len 32768 data: 7 8 8 8 9 8 10 9 10/30 Read block 55 len 36864 11/30 Read block 404 len 12288 12/30 Read block 523 len 20480 13/30 Read block 329 len 4096 14/30 Write block 154 len 16384 data: 7 8 6 6 15/30 Write block 585 len 40960 data: 8 8 9 12 16 15 11 11 9 9 16/30 Read block 59 len 36864 17/30 Write block 509 len 36864 data: 8 11 9 10 9 9 10 10 12 18/30 Write block 42 len 36864 data: 16 13 13 14 13 13 11 10 10 19/30 Flush 20/30 Write block 480 len 28672 data: 7 8 7 8 7 6 4 21/30 Read block 270 len 32768 22/30 Read block 198 len 8192 23/30 Read block 489 len 4096 24/30 Read block 369 len 4096 25/30 Write block 590 len 12288 data: 16 12 12 26/30 Read block 457 len 36864 27/30 Write block 383 len 16384 data: 6 6 7 6 28/30 Write block 558 len 40960 data: 8 8 7 7 7 7 8 6 6 6 29/30 Read block 525 len 32768 30/30 Write block 395 len 28672 data: 9 7 8 7 8 7 9 ---------------------------------------------------------------- Crucible gen:116 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 3 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Write 9 Done Done Done false 1004 Acked Write 1 Done Sent Done false 1005 Acked Read 7 Done Sent Done false 1006 Acked Read 6 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Write 8 Done Sent Done false 1009 Acked Read 9 Done Sent Done false 1010 Acked Read 3 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Read 9 Done Sent Done false 1016 Acked Write 9 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Read 1 Done Sent Done false 1023 Acked Read 1 Done Sent Done false 1024 Acked Write 3 Done Sent Done false 1025 Acked Read 9 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Read 8 Done Sent Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 26 1 28 Done 28 3 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1000)) Some(JobId(1018)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:07.906386063Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1667} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 110 110 192 192 192 F F F 1 020-039 108 108 108 189 189 189 F F F 2 040-059 116 111 116 203 195 203 F F F <--- 3 060-079 116 111 116 203 195 203 F T F <--- 4 080-099 115 115 115 201 201 201 F F F 5 100-119 115 115 115 201 201 201 F F F 6 120-139 115 115 115 201 201 201 F F F 7 140-159 116 108 116 203 188 203 F T F <--- 8 160-179 112 112 112 198 198 198 F F F 9 180-199 112 112 112 198 198 198 F F F 10 200-219 110 110 110 194 194 194 F F F 11 220-239 105 105 105 183 183 183 F F F 12 240-259 115 115 115 201 201 201 F F F 13 260-279 115 115 115 201 201 201 F F F 14 280-299 115 115 115 201 201 201 F F F 15 300-319 116 100 116 203 174 203 F F F <--- 16 320-339 115 115 115 201 201 201 F F F 17 340-359 115 115 115 201 201 201 F F F 18 360-379 115 115 115 201 201 201 F F F 19 380-399 113 113 113 199 199 199 T F T <--- 20 400-419 115 115 115 201 201 201 T F T <--- 21 420-439 115 115 115 201 201 201 F F F 22 440-459 113 113 113 199 199 199 F F F 23 460-479 115 115 115 201 201 201 F F F 24 480-499 115 115 115 201 201 201 T F T <--- 25 500-519 116 114 116 203 200 203 F F F <--- 26 520-539 109 109 109 190 190 190 F F F 27 540-559 112 112 112 196 196 196 T F T <--- 28 560-579 115 115 115 201 201 201 T F T <--- 29 580-599 116 113 116 203 199 203 T F T <--- Max gen: 116, Max flush: 203 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 58 Verifying data now No change after verify Loop: 58 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 110 110 110 192 192 192 F F F 1 020-039 108 108 108 189 189 189 F F F 2 040-059 116 116 116 203 203 203 F F F 3 060-079 116 116 116 203 203 203 F F F 4 080-099 115 115 115 201 201 201 F F F 5 100-119 115 115 115 201 201 201 F F F 6 120-139 115 115 115 201 201 201 F F F 7 140-159 116 116 116 203 203 203 F F F 8 160-179 112 112 112 198 198 198 F F F 9 180-199 112 112 112 198 198 198 F F F 10 200-219 110 110 110 194 194 194 F F F 11 220-239 105 105 105 183 183 183 F F F 12 240-259 115 115 115 201 201 201 F F F 13 260-279 115 115 115 201 201 201 F F F 14 280-299 115 115 115 201 201 201 F F F 15 300-319 116 116 116 203 203 203 F F F 16 320-339 115 115 115 201 201 201 F F F 17 340-359 115 115 115 201 201 201 F F F 18 360-379 115 115 115 201 201 201 F F F 19 380-399 117 117 117 204 204 204 F F F 20 400-419 117 117 117 204 204 204 F F F 21 420-439 115 115 115 201 201 201 F F F 22 440-459 113 113 113 199 199 199 F F F 23 460-479 115 115 115 201 201 201 F F F 24 480-499 117 117 117 204 204 204 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 109 109 109 190 190 190 F F F 27 540-559 117 117 117 204 204 204 F F F 28 560-579 117 117 117 204 204 204 F F F 29 580-599 117 117 117 204 204 204 F F F Max gen: 117, Max flush: 204 Begin loop 59 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.321096042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.366532218Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.367270215Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.367328437Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.367348425Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.367375822Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675} {"msg":"Crucible 0a2fc586-13ea-4d52-8cd0-2090be3a7021 has session id: f254f6a5-5974-4a03-96bc-cb70a3c7be8a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.367853471Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"Upstairs opts: Upstairs UUID: 0a2fc586-13ea-4d52-8cd0-2090be3a7021, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.367979606Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"Crucible stats registered with UUID: 0a2fc586-13ea-4d52-8cd0-2090be3a7021","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.36802014Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:12.368048207Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.3682574Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"io task","client":"0","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368298845Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"io task","client":"2","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368319852Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"io task","client":"1","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368451506Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"task":"crutest"} {"msg":"The guest has requested activation with gen:118","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368476573Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675} {"msg":"0a2fc586-13ea-4d52-8cd0-2090be3a7021 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368545194Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368566061Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"client":"0","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368583409Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"client":"1","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368600257Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"client":"2","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368616615Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"io task","client":"2","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368640242Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"io task","client":"0","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368702104Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"io task","client":"1","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"ds_connection connected from Ok(127.0.0.1:45593)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368749538Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"io task","client":"0","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"ds_connection connected from Ok(127.0.0.1:44209)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368780954Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"io task","client":"2","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"ds_connection connected from Ok(127.0.0.1:61340)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.368810951Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"io task","client":"1","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.370099278Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"client":"0","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.370132184Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"client":"1","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.370651639Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"client":"2","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.370816028Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.370845085Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"[0]R flush_numbers[0..12]: [192, 189, 203, 203, 201, 201, 201, 203, 198, 198, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371209648Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"downstairs","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"[0]R generation[0..12]: [110, 108, 116, 116, 115, 115, 115, 116, 112, 112, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371240185Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"downstairs","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371261142Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"downstairs","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"[1]R flush_numbers[0..12]: [192, 189, 203, 203, 201, 201, 201, 203, 198, 198, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371280699Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"downstairs","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"[1]R generation[0..12]: [110, 108, 116, 116, 115, 115, 115, 116, 112, 112, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371299807Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"downstairs","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371319225Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"downstairs","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"[2]R flush_numbers[0..12]: [192, 189, 203, 203, 201, 201, 201, 203, 198, 198, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371337932Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"downstairs","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"[2]R generation[0..12]: [110, 108, 116, 116, 115, 115, 115, 116, 112, 112, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.37135685Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"downstairs","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371376738Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"downstairs","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"Max found gen is 118","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-11-06T20:16:12.371393175Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"downstairs","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"Generation requested: 118 >= found:118","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371415422Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"downstairs","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"Next flush: 205","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371431471Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"downstairs","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371447549Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"":"downstairs","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371464256Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371482404Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"client":"0","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371499242Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"client":"1","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.37151601Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"client":"2","":"downstairs","session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371533058Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371549256Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"0a2fc586-13ea-4d52-8cd0-2090be3a7021 is now active with session: f254f6a5-5974-4a03-96bc-cb70a3c7be8a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371567643Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371586241Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675,"session_id":"f254f6a5-5974-4a03-96bc-cb70a3c7be8a"} {"msg":"The guest has finished waiting for activation with:118","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371602759Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675} {"msg":"Activated sub_volume 0a2fc586-13ea-4d52-8cd0-2090be3a7021","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:12.371616737Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 585 len 36864 data: 9 9 10 13 17 17 13 13 10 02/30 Read block 475 len 12288 03/30 Write block 484 len 16384 data: 8 7 5 5 04/30 Read block 261 len 20480 05/30 Write block 7 len 20480 data: 7 8 8 8 10 06/30 Read block 409 len 28672 07/30 Flush 08/30 Write block 450 len 32768 data: 15 15 11 11 13 14 14 13 09/30 Write block 447 len 16384 data: 7 10 13 16 10/30 Write block 434 len 16384 data: 10 9 14 13 11/30 Read block 435 len 4096 12/30 Read block 532 len 20480 13/30 Read block 310 len 28672 14/30 Read block 12 len 40960 15/30 Read block 390 len 16384 16/30 Write block 379 len 32768 data: 6 7 7 6 7 7 8 7 17/30 Flush 18/30 Write block 20 len 36864 data: 8 8 11 12 12 13 12 11 11 19/30 Read block 360 len 16384 20/30 Write block 413 len 24576 data: 10 7 7 9 10 14 21/30 Read block 142 len 20480 22/30 Write block 536 len 40960 data: 13 11 12 12 13 13 16 12 10 11 23/30 Write block 38 len 28672 data: 21 18 20 20 17 14 14 24/30 Read block 554 len 16384 25/30 Read block 194 len 32768 26/30 Flush 27/30 Write block 340 len 32768 data: 8 7 6 6 6 6 7 7 28/30 Read block 68 len 8192 29/30 Read block 203 len 40960 30/30 Write block 564 len 4096 data: 9 ---------------------------------------------------------------- Crucible gen:118 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Read 5 Done Done Sent false 1004 Acked Write 5 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 8 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Write 4 Done Done Sent false 1010 Acked Read 1 Done Done Sent false 1011 Acked Read 5 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Read 10 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 9 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Write 6 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Write 10 Done Done Sent false 1022 Acked Write 7 Done Done Sent false 1023 Acked Read 4 Done Done Sent false 1024 Acked Read 8 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 8 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Read 10 Done Done Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 27 29 Done 29 29 3 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:12.392801028Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1675} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 110 205 205 192 F F F <--- 1 020-039 118 118 108 207 207 189 F F F <--- 2 040-059 118 118 116 207 207 203 F F F <--- 3 060-079 116 116 116 203 203 203 F F F 4 080-099 115 115 115 201 201 201 F F F 5 100-119 115 115 115 201 201 201 F F F 6 120-139 115 115 115 201 201 201 F F F 7 140-159 116 116 116 203 203 203 F F F 8 160-179 112 112 112 198 198 198 F F F 9 180-199 112 112 112 198 198 198 F F F 10 200-219 110 110 110 194 194 194 F F F 11 220-239 105 105 105 183 183 183 F F F 12 240-259 115 115 115 201 201 201 F F F 13 260-279 115 115 115 201 201 201 F F F 14 280-299 115 115 115 201 201 201 F F F 15 300-319 116 116 116 203 203 203 F F F 16 320-339 115 115 115 201 201 201 F F F 17 340-359 115 115 115 201 201 201 T T F <--- 18 360-379 118 118 115 206 206 201 F F F <--- 19 380-399 118 118 117 206 206 204 F F F <--- 20 400-419 118 118 117 207 207 204 F F F <--- 21 420-439 118 118 115 206 206 201 F F F <--- 22 440-459 118 118 113 206 206 199 F F F <--- 23 460-479 115 115 115 201 201 201 F F F 24 480-499 118 118 117 205 205 204 F F T <--- 25 500-519 116 116 116 203 203 203 F F F 26 520-539 118 118 109 207 207 190 F F F <--- 27 540-559 118 118 117 207 207 204 F F F <--- 28 560-579 117 117 117 204 204 204 T T F <--- 29 580-599 118 118 117 205 205 204 F F T <--- Max gen: 118, Max flush: 207 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 59 Verifying data now No change after verify Loop: 59 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 205 205 205 F F F 1 020-039 118 118 118 207 207 207 F F F 2 040-059 118 118 118 207 207 207 F F F 3 060-079 116 116 116 203 203 203 F F F 4 080-099 115 115 115 201 201 201 F F F 5 100-119 115 115 115 201 201 201 F F F 6 120-139 115 115 115 201 201 201 F F F 7 140-159 116 116 116 203 203 203 F F F 8 160-179 112 112 112 198 198 198 F F F 9 180-199 112 112 112 198 198 198 F F F 10 200-219 110 110 110 194 194 194 F F F 11 220-239 105 105 105 183 183 183 F F F 12 240-259 115 115 115 201 201 201 F F F 13 260-279 115 115 115 201 201 201 F F F 14 280-299 115 115 115 201 201 201 F F F 15 300-319 116 116 116 203 203 203 F F F 16 320-339 115 115 115 201 201 201 F F F 17 340-359 119 119 119 208 208 208 F F F 18 360-379 118 118 118 206 206 206 F F F 19 380-399 118 118 118 206 206 206 F F F 20 400-419 118 118 118 207 207 207 F F F 21 420-439 118 118 118 206 206 206 F F F 22 440-459 118 118 118 206 206 206 F F F 23 460-479 115 115 115 201 201 201 F F F 24 480-499 118 118 118 205 205 205 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 118 118 118 207 207 207 F F F 27 540-559 118 118 118 207 207 207 F F F 28 560-579 119 119 119 208 208 208 F F F 29 580-599 118 118 118 205 205 205 F F F Max gen: 119, Max flush: 208 Begin loop 60 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.801196323Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:16.844746816Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.845130408Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.876343806Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.876913874Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.876955638Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.876976316Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.877003562Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683} {"msg":"Crucible 2fcbb28b-3684-49a7-bc6c-e31329ebb2d8 has session id: 89d20e5b-49b9-4b09-8fa9-8d6f83ad2385","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.877572441Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"Upstairs opts: Upstairs UUID: 2fcbb28b-3684-49a7-bc6c-e31329ebb2d8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.877629373Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"Crucible stats registered with UUID: 2fcbb28b-3684-49a7-bc6c-e31329ebb2d8","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.877652781Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:16.877669639Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.877864274Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"io task","client":"0","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.877900379Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"io task","client":"2","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.877919157Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"io task","client":"1","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.878068088Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"task":"crutest"} {"msg":"The guest has requested activation with gen:120","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.878094755Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683} {"msg":"2fcbb28b-3684-49a7-bc6c-e31329ebb2d8 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.878168476Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.878189103Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"client":"0","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.878206241Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"client":"1","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.878223119Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"client":"2","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.878240096Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"io task","client":"2","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.878264944Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"io task","client":"0","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.878296329Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"io task","client":"1","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"ds_connection connected from Ok(127.0.0.1:43337)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.878332465Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"io task","client":"2","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"ds_connection connected from Ok(127.0.0.1:34344)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.878502243Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"io task","client":"1","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"ds_connection connected from Ok(127.0.0.1:38907)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.878556547Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"io task","client":"0","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.879652569Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"client":"1","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.879779483Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"client":"0","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.879807549Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"client":"2","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880183292Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880266121Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"[0]R flush_numbers[0..12]: [205, 207, 207, 203, 201, 201, 201, 203, 198, 198, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880395025Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"downstairs","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"[0]R generation[0..12]: [118, 118, 118, 116, 115, 115, 115, 116, 112, 112, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880420342Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"downstairs","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880441249Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"downstairs","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"[1]R flush_numbers[0..12]: [205, 207, 207, 203, 201, 201, 201, 203, 198, 198, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880460997Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"downstairs","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"[1]R generation[0..12]: [118, 118, 118, 116, 115, 115, 115, 116, 112, 112, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880484084Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"downstairs","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880504181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"downstairs","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"[2]R flush_numbers[0..12]: [205, 207, 207, 203, 201, 201, 201, 203, 198, 198, 194, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880523299Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"downstairs","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"[2]R generation[0..12]: [118, 118, 118, 116, 115, 115, 115, 116, 112, 112, 110, 105]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880542157Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"downstairs","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880561684Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"downstairs","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"Max found gen is 120","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880577852Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"Generation requested: 120 >= found:120","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880599979Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"downstairs","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"Next flush: 209","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880616087Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"downstairs","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880632415Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"":"downstairs","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880649293Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880664561Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"client":"0","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880681559Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"client":"1","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880698217Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"client":"2","":"downstairs","session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880715365Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880731503Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"2fcbb28b-3684-49a7-bc6c-e31329ebb2d8 is now active with session: 89d20e5b-49b9-4b09-8fa9-8d6f83ad2385","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:16:16.8807511Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880771968Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683,"session_id":"89d20e5b-49b9-4b09-8fa9-8d6f83ad2385"} {"msg":"The guest has finished waiting for activation with:120","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880788335Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683} {"msg":"Activated sub_volume 2fcbb28b-3684-49a7-bc6c-e31329ebb2d8","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:16.880802504Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 312 len 32768 data: 10 10 11 9 8 8 7 7 02/30 Write block 324 len 32768 data: 9 10 10 9 9 7 7 7 03/30 Read block 156 len 28672 04/30 Write block 422 len 28672 data: 12 10 12 12 12 10 9 05/30 Flush 06/30 Write block 94 len 32768 data: 8 9 9 10 8 9 7 7 07/30 Write block 179 len 20480 data: 16 18 19 19 19 08/30 Read block 511 len 24576 09/30 Read block 133 len 16384 10/30 Write block 265 len 40960 data: 6 12 14 11 11 11 12 12 12 11 11/30 Write block 77 len 32768 data: 8 11 10 10 9 9 8 8 12/30 Read block 188 len 32768 13/30 Write block 219 len 16384 data: 10 11 8 7 14/30 Write block 314 len 16384 data: 12 10 9 9 15/30 Read block 229 len 4096 16/30 Write block 445 len 40960 data: 12 8 8 11 14 17 16 12 12 14 17/30 Write block 298 len 8192 data: 8 9 18/30 Write block 449 len 40960 data: 15 18 17 13 13 15 15 15 14 12 19/30 Read block 514 len 16384 20/30 Write block 158 len 24576 data: 6 9 9 9 10 8 21/30 Write block 400 len 12288 data: 8 10 9 22/30 Write block 287 len 40960 data: 6 5 12 13 11 11 12 11 10 8 23/30 Write block 20 len 32768 data: 9 9 12 13 13 14 13 12 24/30 Write block 337 len 4096 data: 12 25/30 Write block 153 len 24576 data: 6 8 9 7 7 7 26/30 Read block 142 len 8192 27/30 Write block 385 len 4096 data: 9 28/30 Read block 139 len 32768 29/30 Write block 161 len 28672 data: 10 11 9 9 7 8 8 30/30 Read block 262 len 20480 ---------------------------------------------------------------- Crucible gen:120 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 8 Sent Done Done false 1002 Acked Read 7 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 8 Sent Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Read 6 Sent Done Done false 1008 Acked Read 4 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Write 4 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Read 1 Sent Done Done false 1015 Acked Write 10 Sent Done Done false 1016 Acked Write 2 Sent Done Done false 1017 Acked Write 10 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Write 6 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Read 2 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Write 7 Sent Done Done false 1029 Acked Read 5 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 1 30 Done 1 30 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1004)) Some(JobId(1004)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:16.899181338Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1683} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 205 205 205 F F F 1 020-039 118 118 118 207 207 207 F T T <--- 2 040-059 118 118 118 207 207 207 F F F 3 060-079 116 116 116 203 203 203 F T T <--- 4 080-099 115 115 115 201 201 201 F T T <--- 5 100-119 115 115 115 201 201 201 F T T <--- 6 120-139 115 115 115 201 201 201 F F F 7 140-159 116 116 116 203 203 203 F T T <--- 8 160-179 112 112 112 198 198 198 F T T <--- 9 180-199 112 112 112 198 198 198 F T T <--- 10 200-219 110 110 110 194 194 194 F T T <--- 11 220-239 105 105 105 183 183 183 F T T <--- 12 240-259 115 115 115 201 201 201 F F F 13 260-279 115 115 115 201 201 201 F T T <--- 14 280-299 115 115 115 201 201 201 F T T <--- 15 300-319 116 120 120 203 209 209 T T T <--- 16 320-339 115 120 120 201 209 209 F T T <--- 17 340-359 119 119 119 208 208 208 F F F 18 360-379 118 118 118 206 206 206 F F F 19 380-399 118 118 118 206 206 206 F T T <--- 20 400-419 118 118 118 207 207 207 F T T <--- 21 420-439 118 120 120 206 209 209 F F F <--- 22 440-459 118 118 118 206 206 206 F T T <--- 23 460-479 115 115 115 201 201 201 F F F 24 480-499 118 118 118 205 205 205 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 118 118 118 207 207 207 F F F 27 540-559 118 118 118 207 207 207 F F F 28 560-579 119 119 119 208 208 208 F F F 29 580-599 118 118 118 205 205 205 F F F Max gen: 120, Max flush: 209 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 60 Verifying data now No change after verify Loop: 60 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 205 205 205 F F F 1 020-039 121 121 121 210 210 210 F F F 2 040-059 118 118 118 207 207 207 F F F 3 060-079 121 121 121 210 210 210 F F F 4 080-099 121 121 121 210 210 210 F F F 5 100-119 121 121 121 210 210 210 F F F 6 120-139 115 115 115 201 201 201 F F F 7 140-159 121 121 121 210 210 210 F F F 8 160-179 121 121 121 210 210 210 F F F 9 180-199 121 121 121 210 210 210 F F F 10 200-219 121 121 121 210 210 210 F F F 11 220-239 121 121 121 210 210 210 F F F 12 240-259 115 115 115 201 201 201 F F F 13 260-279 121 121 121 210 210 210 F F F 14 280-299 121 121 121 210 210 210 F F F 15 300-319 121 121 121 210 210 210 F F F 16 320-339 121 121 121 210 210 210 F F F 17 340-359 119 119 119 208 208 208 F F F 18 360-379 118 118 118 206 206 206 F F F 19 380-399 121 121 121 210 210 210 F F F 20 400-419 121 121 121 210 210 210 F F F 21 420-439 120 120 120 209 209 209 F F F 22 440-459 121 121 121 210 210 210 F F F 23 460-479 115 115 115 201 201 201 F F F 24 480-499 118 118 118 205 205 205 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 118 118 118 207 207 207 F F F 27 540-559 118 118 118 207 207 207 F F F 28 560-579 119 119 119 208 208 208 F F F 29 580-599 118 118 118 205 205 205 F F F Max gen: 121, Max flush: 210 Begin loop 61 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.364836479Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.410211513Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.410807198Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.410855541Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.410873059Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.410895967Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691} {"msg":"Crucible 963e1e43-6eda-4767-a4d3-ef7cfbc46b44 has session id: df3bda21-c6ec-451f-81b2-ef52d221b5a0","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.411405322Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Upstairs opts: Upstairs UUID: 963e1e43-6eda-4767-a4d3-ef7cfbc46b44, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.411448167Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Crucible stats registered with UUID: 963e1e43-6eda-4767-a4d3-ef7cfbc46b44","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.411468264Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:21.411488012Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.411682168Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"io task","client":"0","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.411719733Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"io task","client":"2","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.41173871Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"io task","client":"1","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.411920338Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"task":"crutest"} {"msg":"The guest has requested activation with gen:122","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.411948674Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691} {"msg":"963e1e43-6eda-4767-a4d3-ef7cfbc46b44 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.412025274Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.412047391Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"client":"0","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.412064859Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"client":"1","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.412081547Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"client":"2","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.412100515Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"io task","client":"2","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.412128261Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"io task","client":"0","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.412173295Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"io task","client":"1","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"ds_connection connected from Ok(127.0.0.1:46556)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.412224989Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"io task","client":"2","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"ds_connection connected from Ok(127.0.0.1:58717)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.412258025Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"io task","client":"1","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"ds_connection connected from Ok(127.0.0.1:64426)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.412290461Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"io task","client":"0","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.41348922Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"client":"0","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.413538774Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"client":"2","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.414169294Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.414233206Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.414829421Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"client":"1","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"[0]R flush_numbers[0..12]: [205, 210, 207, 210, 210, 210, 201, 210, 210, 210, 210, 210]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415334147Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"downstairs","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"[0]R generation[0..12]: [118, 121, 118, 121, 121, 121, 115, 121, 121, 121, 121, 121]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415365473Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"downstairs","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.41539124Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"downstairs","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"[1]R flush_numbers[0..12]: [205, 210, 207, 210, 210, 210, 201, 210, 210, 210, 210, 210]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415415687Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"downstairs","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"[1]R generation[0..12]: [118, 121, 118, 121, 121, 121, 115, 121, 121, 121, 121, 121]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415439284Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"downstairs","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415464131Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"downstairs","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"[2]R flush_numbers[0..12]: [205, 210, 207, 210, 210, 210, 201, 210, 210, 210, 210, 210]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415488188Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"downstairs","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"[2]R generation[0..12]: [118, 121, 118, 121, 121, 121, 115, 121, 121, 121, 121, 121]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415511765Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"downstairs","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Wait for a query_work_queue command to finish before sending IO [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415540651Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"downstairs","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Max found gen is 122","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415565848Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"downstairs","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Generation requested: 122 >= found:122","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415587795Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"downstairs","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Next flush: 211","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415608103Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"downstairs","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.4156284Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"":"downstairs","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415649517Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415668515Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"client":"0","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415689843Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"client":"1","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.41571105Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"client":"2","":"downstairs","session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415736966Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415757524Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"963e1e43-6eda-4767-a4d3-ef7cfbc46b44 is now active with session: df3bda21-c6ec-451f-81b2-ef52d221b5a0","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415780101Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415798939Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691,"session_id":"df3bda21-c6ec-451f-81b2-ef52d221b5a0"} {"msg":"The guest has finished waiting for activation with:122","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415819086Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691} {"msg":"Activated sub_volume 963e1e43-6eda-4767-a4d3-ef7cfbc46b44","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:21.415836514Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 591 len 20480 data: 14 14 11 10 10 02/30 Flush 03/30 Write block 428 len 16384 data: 10 9 10 8 04/30 Flush 05/30 Write block 54 len 8192 data: 6 5 06/30 Read block 544 len 36864 07/30 Read block 59 len 28672 08/30 Write block 437 len 36864 data: 14 12 14 11 12 11 11 13 13 09/30 Read block 183 len 8192 10/30 Read block 398 len 40960 11/30 Write block 282 len 20480 data: 7 7 6 6 6 12/30 Read block 525 len 24576 13/30 Flush 14/30 Write block 404 len 16384 data: 10 10 9 9 15/30 Write block 409 len 4096 data: 11 16/30 Read block 34 len 28672 17/30 Read block 141 len 40960 18/30 Write block 232 len 36864 data: 10 10 8 10 10 8 9 10 9 19/30 Read block 565 len 32768 20/30 Write block 56 len 16384 data: 4 5 7 5 21/30 Read block 119 len 4096 22/30 Write block 238 len 28672 data: 10 11 10 10 13 15 15 23/30 Flush 24/30 Write block 582 len 28672 data: 6 6 5 10 10 11 14 25/30 Read block 88 len 12288 26/30 Write block 394 len 8192 data: 10 10 27/30 Write block 480 len 24576 data: 8 9 8 9 9 8 28/30 Read block 249 len 24576 29/30 Read block 171 len 20480 30/30 Write block 420 len 4096 data: 13 ---------------------------------------------------------------- Crucible gen:122 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 4 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 2 Done Sent Done false 1005 Acked Read 9 Done Sent Done false 1006 Acked Read 7 Done Sent Done false 1007 Acked Write 9 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Read 10 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Write 1 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Read 10 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Read 8 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Read 1 Done Sent Done false 1021 Acked Write 7 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 7 Done Sent Done false 1024 Acked Read 3 Done Sent Done false 1025 Acked Write 2 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Read 5 Done Sent Done false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 1 31 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:21.435024427Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1691} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 205 205 205 F F F 1 020-039 121 121 121 210 210 210 F F F 2 040-059 122 118 122 214 207 214 F F F <--- 3 060-079 121 121 121 210 210 210 F F F 4 080-099 121 121 121 210 210 210 F F F 5 100-119 121 121 121 210 210 210 F F F 6 120-139 115 115 115 201 201 201 F F F 7 140-159 121 121 121 210 210 210 F F F 8 160-179 121 121 121 210 210 210 F F F 9 180-199 121 121 121 210 210 210 F F F 10 200-219 121 121 121 210 210 210 F F F 11 220-239 122 121 122 214 210 214 F F F <--- 12 240-259 122 115 122 214 201 214 F F F <--- 13 260-279 121 121 121 210 210 210 F F F 14 280-299 122 121 122 213 210 213 F F F <--- 15 300-319 121 121 121 210 210 210 F F F 16 320-339 121 121 121 210 210 210 F F F 17 340-359 119 119 119 208 208 208 F F F 18 360-379 118 118 118 206 206 206 F F F 19 380-399 121 121 121 210 210 210 T F T <--- 20 400-419 122 121 122 214 210 214 F F F <--- 21 420-439 122 120 122 213 209 213 T F T <--- 22 440-459 122 121 122 213 210 213 F F F <--- 23 460-479 115 115 115 201 201 201 F F F 24 480-499 118 118 118 205 205 205 T F T <--- 25 500-519 116 116 116 203 203 203 F F F 26 520-539 118 118 118 207 207 207 F F F 27 540-559 118 118 118 207 207 207 F F F 28 560-579 119 119 119 208 208 208 F F F 29 580-599 122 118 122 211 205 211 T T T <--- Max gen: 122, Max flush: 214 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 61 Verifying data now No change after verify Loop: 61 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 205 205 205 F F F 1 020-039 121 121 121 210 210 210 F F F 2 040-059 122 122 122 214 214 214 F F F 3 060-079 121 121 121 210 210 210 F F F 4 080-099 121 121 121 210 210 210 F F F 5 100-119 121 121 121 210 210 210 F F F 6 120-139 115 115 115 201 201 201 F F F 7 140-159 121 121 121 210 210 210 F F F 8 160-179 121 121 121 210 210 210 F F F 9 180-199 121 121 121 210 210 210 F F F 10 200-219 121 121 121 210 210 210 F F F 11 220-239 122 122 122 214 214 214 F F F 12 240-259 122 122 122 214 214 214 F F F 13 260-279 121 121 121 210 210 210 F F F 14 280-299 122 122 122 213 213 213 F F F 15 300-319 121 121 121 210 210 210 F F F 16 320-339 121 121 121 210 210 210 F F F 17 340-359 119 119 119 208 208 208 F F F 18 360-379 118 118 118 206 206 206 F F F 19 380-399 123 123 123 215 215 215 F F F 20 400-419 122 122 122 214 214 214 F F F 21 420-439 123 123 123 215 215 215 F F F 22 440-459 122 122 122 213 213 213 F F F 23 460-479 115 115 115 201 201 201 F F F 24 480-499 123 123 123 215 215 215 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 118 118 118 207 207 207 F F F 27 540-559 118 118 118 207 207 207 F F F 28 560-579 119 119 119 208 208 208 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 123, Max flush: 215 Begin loop 62 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.827060712Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:25.870831687Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.871225947Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.902757435Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.903412312Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.903460606Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.903477484Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.903499691Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699} {"msg":"Crucible 6c598f35-92d9-4d80-abc3-c4ac1bddcdb2 has session id: 5653e85b-daa2-4a28-9aac-9caaade9c1b3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904114694Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"Upstairs opts: Upstairs UUID: 6c598f35-92d9-4d80-abc3-c4ac1bddcdb2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904168987Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"Crucible stats registered with UUID: 6c598f35-92d9-4d80-abc3-c4ac1bddcdb2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904191764Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:25.904208972Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.90438569Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"io task","client":"0","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904424545Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"io task","client":"2","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904443382Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"io task","client":"1","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.90462176Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"task":"crutest"} {"msg":"The guest has requested activation with gen:124","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904642977Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699} {"msg":"6c598f35-92d9-4d80-abc3-c4ac1bddcdb2 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904713708Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904734426Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"client":"0","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904751763Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"client":"1","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904768512Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"client":"2","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904790069Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"io task","client":"2","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904820805Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"io task","client":"0","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904852351Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"io task","client":"1","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"ds_connection connected from Ok(127.0.0.1:33908)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904922542Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"io task","client":"2","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"ds_connection connected from Ok(127.0.0.1:38154)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.904969676Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"io task","client":"1","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"ds_connection connected from Ok(127.0.0.1:33181)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.905086532Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"io task","client":"0","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.906160026Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"client":"2","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.906269552Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"client":"1","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.906319206Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"client":"0","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.906759201Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.906891954Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"[0]R flush_numbers[0..12]: [205, 210, 214, 210, 210, 210, 201, 210, 210, 210, 210, 214]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.906940788Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"downstairs","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"[0]R generation[0..12]: [118, 121, 122, 121, 121, 121, 115, 121, 121, 121, 121, 122]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.906963405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"downstairs","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.906983732Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"downstairs","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"[1]R flush_numbers[0..12]: [205, 210, 214, 210, 210, 210, 201, 210, 210, 210, 210, 214]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.90700295Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"downstairs","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"[1]R generation[0..12]: [118, 121, 122, 121, 121, 121, 115, 121, 121, 121, 121, 122]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907022128Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"downstairs","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907041945Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"downstairs","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"[2]R flush_numbers[0..12]: [205, 210, 214, 210, 210, 210, 201, 210, 210, 210, 210, 214]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907061113Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"downstairs","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"[2]R generation[0..12]: [118, 121, 122, 121, 121, 121, 115, 121, 121, 121, 121, 122]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.90708008Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"downstairs","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907099888Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"downstairs","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"Max found gen is 124","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907116236Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"downstairs","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"Generation requested: 124 >= found:124","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907133074Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"downstairs","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"Next flush: 216","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907149132Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"Wait for a query_work_queue command to finish before sending IO downstairs","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907172059Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"":"downstairs","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907189316Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907206584Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"client":"0","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907223692Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"client":"1","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.90724068Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"client":"2","":"downstairs","session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907257918Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907274106Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"6c598f35-92d9-4d80-abc3-c4ac1bddcdb2 is now active with session: 5653e85b-daa2-4a28-9aac-9caaade9c1b3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907292063Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907307292Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699,"session_id":"5653e85b-daa2-4a28-9aac-9caaade9c1b3"} {"msg":"The guest has finished waiting for activation with:124","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.90732309Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699} {"msg":"Activated sub_volume 6c598f35-92d9-4d80-abc3-c4ac1bddcdb2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:25.907336828Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 367 len 20480 data: 7 7 6 5 6 02/30 Write block 224 len 40960 data: 7 6 5 7 8 10 10 10 11 11 03/30 Write block 542 len 12288 data: 17 13 11 04/30 Read block 537 len 8192 05/30 Read block 50 len 36864 06/30 Write block 84 len 32768 data: 9 7 8 8 9 11 10 10 07/30 Write block 52 len 8192 data: 7 6 08/30 Write block 310 len 12288 data: 10 11 11 09/30 Write block 539 len 16384 data: 13 14 14 18 10/30 Read block 576 len 24576 11/30 Read block 529 len 36864 12/30 Read block 373 len 4096 13/30 Flush 14/30 Flush 15/30 Write block 22 len 20480 data: 13 14 14 15 14 16/30 Read block 47 len 24576 17/30 Write block 109 len 12288 data: 9 10 12 18/30 Write block 202 len 12288 data: 11 11 11 19/30 Read block 586 len 8192 20/30 Write block 381 len 32768 data: 8 7 8 8 10 8 6 7 21/30 Write block 223 len 8192 data: 10 8 22/30 Flush 23/30 Write block 241 len 40960 data: 11 14 16 16 15 14 12 12 11 11 24/30 Read block 409 len 4096 25/30 Flush 26/30 Write block 19 len 4096 data: 5 27/30 Write block 78 len 20480 data: 12 11 11 10 10 28/30 Write block 191 len 4096 data: 8 29/30 Write block 210 len 16384 data: 5 4 5 6 30/30 Read block 513 len 12288 ---------------------------------------------------------------- Crucible gen:124 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 10 Sent Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Read 2 Sent Done Done false 1004 Acked Read 9 Sent Done Done false 1005 Acked Write 8 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Write 3 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Read 6 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Read 1 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Write 3 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Write 4 Sent Done Done false 1029 Acked Read 3 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 1 30 Done 1 30 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:25.924729917Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1699} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 118 118 118 205 205 205 F T T <--- 1 020-039 121 124 124 210 218 218 F F F <--- 2 040-059 122 124 124 214 216 216 F F F <--- 3 060-079 121 121 121 210 210 210 F T T <--- 4 080-099 121 124 124 210 216 216 F T T <--- 5 100-119 121 124 124 210 218 218 F F F <--- 6 120-139 115 115 115 201 201 201 F F F 7 140-159 121 121 121 210 210 210 F F F 8 160-179 121 121 121 210 210 210 F F F 9 180-199 121 121 121 210 210 210 F T T <--- 10 200-219 121 124 124 210 218 218 F T T <--- 11 220-239 122 124 124 214 218 218 F F F <--- 12 240-259 122 124 124 214 219 219 F F F <--- 13 260-279 121 121 121 210 210 210 F F F 14 280-299 122 122 122 213 213 213 F F F 15 300-319 121 124 124 210 216 216 F F F <--- 16 320-339 121 121 121 210 210 210 F F F 17 340-359 119 119 119 208 208 208 F F F 18 360-379 118 124 124 206 216 216 T F F <--- 19 380-399 123 124 124 215 218 218 F F F <--- 20 400-419 122 122 122 214 214 214 F F F 21 420-439 123 123 123 215 215 215 F F F 22 440-459 122 122 122 213 213 213 F F F 23 460-479 115 115 115 201 201 201 F F F 24 480-499 123 123 123 215 215 215 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 118 124 124 207 216 216 F F F <--- 27 540-559 118 124 124 207 216 216 F F F <--- 28 560-579 119 119 119 208 208 208 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 124, Max flush: 219 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 62 Verifying data now No change after verify Loop: 62 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 220 220 220 F F F 1 020-039 124 124 124 218 218 218 F F F 2 040-059 124 124 124 216 216 216 F F F 3 060-079 125 125 125 220 220 220 F F F 4 080-099 125 125 125 220 220 220 F F F 5 100-119 124 124 124 218 218 218 F F F 6 120-139 115 115 115 201 201 201 F F F 7 140-159 121 121 121 210 210 210 F F F 8 160-179 121 121 121 210 210 210 F F F 9 180-199 125 125 125 220 220 220 F F F 10 200-219 125 125 125 220 220 220 F F F 11 220-239 124 124 124 218 218 218 F F F 12 240-259 124 124 124 219 219 219 F F F 13 260-279 121 121 121 210 210 210 F F F 14 280-299 122 122 122 213 213 213 F F F 15 300-319 124 124 124 216 216 216 F F F 16 320-339 121 121 121 210 210 210 F F F 17 340-359 119 119 119 208 208 208 F F F 18 360-379 124 124 124 216 216 216 F F F 19 380-399 124 124 124 218 218 218 F F F 20 400-419 122 122 122 214 214 214 F F F 21 420-439 123 123 123 215 215 215 F F F 22 440-459 122 122 122 213 213 213 F F F 23 460-479 115 115 115 201 201 201 F F F 24 480-499 123 123 123 215 215 215 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 124 124 124 216 216 216 F F F 27 540-559 124 124 124 216 216 216 F F F 28 560-579 119 119 119 208 208 208 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 125, Max flush: 220 Begin loop 63 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.366698711Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:30.410265372Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.410699108Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.442077924Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.442644183Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.442691447Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.442707615Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.442734522Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707} {"msg":"Crucible 2cd6a4bd-8a92-4c4c-84f4-e6c94168fefb has session id: 75cf7871-c8db-499d-a27e-97179796a0b6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.443345685Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"Upstairs opts: Upstairs UUID: 2cd6a4bd-8a92-4c4c-84f4-e6c94168fefb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.443388569Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"Crucible stats registered with UUID: 2cd6a4bd-8a92-4c4c-84f4-e6c94168fefb","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.443408377Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:30.443424965Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.44361688Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"io task","client":"0","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.443654966Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"io task","client":"1","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.443674153Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"io task","client":"2","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.443889376Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"task":"crutest"} {"msg":"The guest has requested activation with gen:126","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.443912973Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707} {"msg":"2cd6a4bd-8a92-4c4c-84f4-e6c94168fefb active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.443990583Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.444011161Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"client":"0","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.444028239Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"client":"1","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.444045106Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"client":"2","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.444071093Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"io task","client":"2","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.444170711Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"io task","client":"0","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.444203477Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"io task","client":"1","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"ds_connection connected from Ok(127.0.0.1:60157)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.444238922Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"io task","client":"1","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"ds_connection connected from Ok(127.0.0.1:39515)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.444270858Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"io task","client":"2","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"ds_connection connected from Ok(127.0.0.1:60883)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.444301474Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"io task","client":"0","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.445544597Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"client":"1","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.445580573Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"client":"2","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.44560091Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"client":"0","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446138683Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446167419Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"[0]R flush_numbers[0..12]: [220, 218, 216, 220, 220, 218, 201, 210, 210, 220, 220, 218]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446212843Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"downstairs","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"[0]R generation[0..12]: [125, 124, 124, 125, 125, 124, 115, 121, 121, 125, 125, 124]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446235101Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"downstairs","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446255218Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"downstairs","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"[1]R flush_numbers[0..12]: [220, 218, 216, 220, 220, 218, 201, 210, 210, 220, 220, 218]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446274396Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"downstairs","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"[1]R generation[0..12]: [125, 124, 124, 125, 125, 124, 115, 121, 121, 125, 125, 124]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446300352Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"downstairs","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446338908Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"downstairs","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"[2]R flush_numbers[0..12]: [220, 218, 216, 220, 220, 218, 201, 210, 210, 220, 220, 218]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446367174Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"downstairs","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"[2]R generation[0..12]: [125, 124, 124, 125, 125, 124, 115, 121, 121, 125, 125, 124]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446393071Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"downstairs","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446427436Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"downstairs","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"Max found gen is 126","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446451473Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"downstairs","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg"Wait for a query_work_queue command to finish before sending IO :"Generation requested: 126 >= found:126","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446487619Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"downstairs","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"Next flush: 221","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446519115Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"downstairs","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446539992Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"":"downstairs","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.4465609Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446586676Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"client":"0","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446617902Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"client":"1","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.44663894Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"client":"2","":"downstairs","session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446662607Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446690813Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"2cd6a4bd-8a92-4c4c-84f4-e6c94168fefb is now active with session: 75cf7871-c8db-499d-a27e-97179796a0b6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.44672036Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.446742617Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707,"session_id":"75cf7871-c8db-499d-a27e-97179796a0b6"} {"msg":"The guest has finished waiting for activation with:126","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:16:30.446770343Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707} {"msg":"Activated sub_volume 2cd6a4bd-8a92-4c4c-84f4-e6c94168fefb","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:30.44679375Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 319 len 12288 data: 8 7 9 02/30 Read block 403 len 32768 03/30 Read block 140 len 28672 04/30 Read block 488 len 20480 05/30 Read block 545 len 40960 06/30 Read block 128 len 40960 07/30 Read block 551 len 20480 08/30 Read block 315 len 40960 09/30 Write block 377 len 40960 data: 6 6 7 8 9 8 9 9 11 9 10/30 Write block 363 len 12288 data: 7 7 8 11/30 Read block 415 len 4096 12/30 Read block 348 len 36864 13/30 Flush 14/30 Write block 478 len 32768 data: 10 8 9 10 9 10 10 9 15/30 Write block 318 len 36864 data: 8 9 8 10 7 8 10 11 11 16/30 Read block 330 len 24576 17/30 Read block 531 len 4096 18/30 Read block 480 len 28672 19/30 Flush 20/30 Write block 438 len 20480 data: 13 15 12 13 12 21/30 Read block 376 len 28672 22/30 Write block 543 len 8192 data: 14 12 23/30 Read block 82 len 32768 24/30 Read block 469 len 32768 25/30 Read block 572 len 24576 26/30 Read block 378 len 4096 27/30 Read block 494 len 36864 28/30 Write block 52 len 40960 data: 8 7 7 6 5 6 8 6 6 4 29/30 Write block 68 len 16384 data: 9 8 7 7 30/30 Read block 383 len 12288 ---------------------------------------------------------------- Crucible gen:126 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Sent Done Done false 1001 Acked Read 8 Sent Done Done false 1002 Acked Read 7 Sent Done Done false 1003 Acked Read 5 Sent Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Read 10 Sent Done Done false 1006 Acked Read 5 Sent Done Done false 1007 Acked Read 10 Sent Done Done false 1008 Acked Write 10 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Read 1 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Read 1 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Read 8 Sent Done Done false 1023 Acked Read 8 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Read 1 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Write 10 Sent Done Done false 1028 Acked Write 4 Sent Done Sent false 1029 Acked Read 3 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 2 32 Done 0 30 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1018)) Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:30.467419302Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1707} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 220 220 220 F F F 1 020-039 124 124 124 218 218 218 F F F 2 040-059 124 124 124 216 216 216 F T T <--- 3 060-079 125 125 125 220 220 220 F T T <--- 4 080-099 125 125 125 220 220 220 F F F 5 100-119 124 124 124 218 218 218 F F F 6 120-139 115 115 115 201 201 201 F F F 7 140-159 121 121 121 210 210 210 F F F 8 160-179 121 121 121 210 210 210 F F F 9 180-199 125 125 125 220 220 220 F F F 10 200-219 125 125 125 220 220 220 F F F 11 220-239 124 124 124 218 218 218 F F F 12 240-259 124 124 124 219 219 219 F F F 13 260-279 121 121 121 210 210 210 F F F 14 280-299 122 122 122 213 213 213 F F F 15 300-319 124 126 126 216 222 222 F F F <--- 16 320-339 121 126 126 210 222 222 F F F <--- 17 340-359 119 119 119 208 208 208 F F F 18 360-379 124 126 126 216 221 221 F F F <--- 19 380-399 124 126 126 218 221 221 F F F <--- 20 400-419 122 122 122 214 214 214 F F F 21 420-439 123 123 123 215 215 215 F T T <--- 22 440-459 122 122 122 213 213 213 F T T <--- 23 460-479 115 126 126 201 222 222 F F F <--- 24 480-499 123 126 126 215 222 222 F F F <--- 25 500-519 116 116 116 203 203 203 F F F 26 520-539 124 124 124 216 216 216 F F F 27 540-559 124 124 124 216 216 216 F T T <--- 28 560-579 119 119 119 208 208 208 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 126, Max flush: 222 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 63 Verifying data now No change after verify Loop: 63 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 220 220 220 F F F 1 020-039 124 124 124 218 218 218 F F F 2 040-059 127 127 127 223 223 223 F F F 3 060-079 127 127 127 223 223 223 F F F 4 080-099 125 125 125 220 220 220 F F F 5 100-119 124 124 124 218 218 218 F F F 6 120-139 115 115 115 201 201 201 F F F 7 140-159 121 121 121 210 210 210 F F F 8 160-179 121 121 121 210 210 210 F F F 9 180-199 125 125 125 220 220 220 F F F 10 200-219 125 125 125 220 220 220 F F F 11 220-239 124 124 124 218 218 218 F F F 12 240-259 124 124 124 219 219 219 F F F 13 260-279 121 121 121 210 210 210 F F F 14 280-299 122 122 122 213 213 213 F F F 15 300-319 126 126 126 222 222 222 F F F 16 320-339 126 126 126 222 222 222 F F F 17 340-359 119 119 119 208 208 208 F F F 18 360-379 126 126 126 221 221 221 F F F 19 380-399 126 126 126 221 221 221 F F F 20 400-419 122 122 122 214 214 214 F F F 21 420-439 127 127 127 223 223 223 F F F 22 440-459 127 127 127 223 223 223 F F F 23 460-479 126 126 126 222 222 222 F F F 24 480-499 126 126 126 222 222 222 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 124 124 124 216 216 216 F F F 27 540-559 127 127 127 223 223 223 F F F 28 560-579 119 119 119 208 208 208 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 127, Max flush: 223 Begin loop 64 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.89633653Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.941839717Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.942500954Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.942549767Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.942566025Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.942589533Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715} {"msg":"Crucible 43bd59e5-b699-467c-8a97-2103431437b7 has session id: 55c19d1d-268f-48eb-87a4-29d0a13ee7ac","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943170739Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"Upstairs opts: Upstairs UUID: 43bd59e5-b699-467c-8a97-2103431437b7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943229992Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"Crucible stats registered with UUID: 43bd59e5-b699-467c-8a97-2103431437b7","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943260838Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:34.943282745Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943490489Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"io task","client":"0","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943536883Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"io task","client":"2","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.94355732Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"io task","client":"1","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943704972Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"task":"crutest"} {"msg":"The guest has requested activation with gen:128","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943732249Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715} {"msg":"43bd59e5-b699-467c-8a97-2103431437b7 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943820407Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943849274Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"client":"0","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943868331Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"client":"1","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943885269Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"client":"2","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943904847Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"io task","client":"2","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943934003Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"io task","client":"0","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.943966199Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"io task","client":"1","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:58557)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.944076575Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"io task","client":"0","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:36931)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.94411841Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"io task","client":"2","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:50535)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.944225736Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"io task","client":"1","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.9453031Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"client":"1","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.94538534Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"client":"0","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.945567707Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"client":"2","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.945852751Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.945928572Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"[0]R flush_numbers[0..12]: [220, 218, 223, 223, 220, 218, 201, 210, 210, 220, 220, 218]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946124847Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"downstairs","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"[0]R generation[0..12]: [125, 124, 127, 127, 125, 124, 115, 121, 121, 125, 125, 124]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946148754Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"downstairs","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946168911Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"downstairs","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"[1]R flush_numbers[0..12]: [220, 218, 223, 223, 220, 218, 201, 210, 210, 220, 220, 218]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946187999Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"downstairs","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"[1]R generation[0..12]: [125, 124, 127, 127, 125, 124, 115, 121, 121, 125, 125, 124]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946207137Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"downstairs","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946226724Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"downstairs","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"[2]R flush_numbers[0..12]: [220, 218, 223, 223, 220, 218, 201, 210, 210, 220, 220, 218]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946245682Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"downstairs","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"[2]R generation[0..12]: [125, 124, 127, 127, 125, 124, 115, 121, 121, 125, 125, 124]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946264239Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"downstairs","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946283867Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"downstairs","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"Max found gen is 128","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946299955Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"downstairs","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"Generation requested: 128 >= found:128","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946317103Z","Wait for a query_work_queue command to finish before sending IO hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"downstairs","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"Next flush: 224","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.94634398Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"downstairs","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946360567Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"":"downstairs","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946377615Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946393753Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"client":"0","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946410661Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"client":"1","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946427189Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"client":"2","":"downstairs","session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946443947Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946459945Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"43bd59e5-b699-467c-8a97-2103431437b7 is now active with session: 55c19d1d-268f-48eb-87a4-29d0a13ee7ac","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946477772Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946493111Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715,"session_id":"55c19d1d-268f-48eb-87a4-29d0a13ee7ac"} {"msg":"The guest has finished waiting for activation with:128","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2025-11-06T20:16:34.946509219Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715} {"msg":"Activated sub_volume 43bd59e5-b699-467c-8a97-2103431437b7","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:34.946526326Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 309 len 24576 data: 10 11 12 12 11 13 02/30 Write block 75 len 40960 data: 6 8 9 13 12 12 11 11 9 10 03/30 Write block 290 len 36864 data: 14 12 12 13 12 11 9 7 9 04/30 Write block 157 len 20480 data: 8 8 10 10 11 05/30 Read block 154 len 8192 06/30 Write block 352 len 24576 data: 7 9 8 8 8 9 07/30 Flush 08/30 Write block 565 len 36864 data: 7 7 7 4 3 4 5 4 5 09/30 Write block 81 len 16384 data: 12 12 10 11 10/30 Write block 568 len 32768 data: 5 4 5 6 5 6 5 7 11/30 Write block 392 len 12288 data: 12 11 11 12/30 Write block 220 len 16384 data: 12 9 8 11 13/30 Write block 110 len 36864 data: 11 13 10 11 14 12 19 16 15 14/30 Read block 321 len 32768 15/30 Read block 39 len 36864 16/30 Write block 173 len 4096 data: 5 17/30 Write block 189 len 20480 data: 9 7 9 9 8 18/30 Write block 335 len 24576 data: 12 11 13 12 11 9 19/30 Write block 477 len 32768 data: 9 11 9 10 11 10 11 11 20/30 Write block 327 len 40960 data: 10 10 8 8 8 9 11 11 13 12 21/30 Write block 45 len 40960 data: 15 14 14 12 11 11 9 9 8 8 22/30 Read block 83 len 28672 23/30 Read block 190 len 20480 24/30 Write block 124 len 16384 data: 10 10 9 11 25/30 Read block 316 len 36864 26/30 Read block 36 len 36864 27/30 Read block 503 len 8192 28/30 Read block 146 len 40960 29/30 Read block 148 len 36864 30/30 Write block 209 len 12288 data: 5 6 5 ---------------------------------------------------------------- Crucible gen:128 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 10 Done Done Sent false 1002 Acked Write 9 Done Done Sent false 1003 Acked Write 5 Done Done Sent false 1004 Acked Read 2 Done Done Sent false 1005 Acked Write 6 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Write 8 Done Done Sent false 1010 Acked Write 3 Done Done Sent false 1011 Acked Write 4 Done Done Sent false 1012 Acked Write 9 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Write 1 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Write 6 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Read 5 Done Done Sent false 1023 Acked Write 4 Done Done Sent false 1024 Acked Read 9 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Read 2 Done Done Sent false 1027 Acked Read 10 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1006)) Some(JobId(1006)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:34.964876634Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1715} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 220 220 220 F F F 1 020-039 124 124 124 218 218 218 F F F 2 040-059 127 127 127 223 223 223 T T F <--- 3 060-079 128 128 127 224 224 223 F F F <--- 4 080-099 128 128 125 224 224 220 T T F <--- 5 100-119 124 124 124 218 218 218 T T F <--- 6 120-139 115 115 115 201 201 201 T T F <--- 7 140-159 128 128 121 224 224 210 F F F <--- 8 160-179 128 128 121 224 224 210 T T F <--- 9 180-199 125 125 125 220 220 220 T T F <--- 10 200-219 125 125 125 220 220 220 T T F <--- 11 220-239 124 124 124 218 218 218 T T F <--- 12 240-259 124 124 124 219 219 219 F F F 13 260-279 121 121 121 210 210 210 F F F 14 280-299 128 128 122 224 224 213 F F F <--- 15 300-319 128 128 126 224 224 222 F F T <--- 16 320-339 126 126 126 222 222 222 T T F <--- 17 340-359 128 128 119 224 224 208 T T F <--- 18 360-379 126 126 126 221 221 221 F F F 19 380-399 126 126 126 221 221 221 T T F <--- 20 400-419 122 122 122 214 214 214 F F F 21 420-439 127 127 127 223 223 223 F F F 22 440-459 127 127 127 223 223 223 F F F 23 460-479 126 126 126 222 222 222 T T F <--- 24 480-499 126 126 126 222 222 222 T T F <--- 25 500-519 116 116 116 203 203 203 F F F 26 520-539 124 124 124 216 216 216 F F F 27 540-559 127 127 127 223 223 223 F F F 28 560-579 119 119 119 208 208 208 T T F <--- 29 580-599 123 123 123 215 215 215 F F F Max gen: 128, Max flush: 224 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 64 Verifying data now No change after verify Loop: 64 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 220 220 220 F F F 1 020-039 124 124 124 218 218 218 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 128 128 128 224 224 224 F F F 4 080-099 129 129 129 225 225 225 F F F 5 100-119 129 129 129 225 225 225 F F F 6 120-139 129 129 129 225 225 225 F F F 7 140-159 128 128 128 224 224 224 F F F 8 160-179 129 129 129 225 225 225 F F F 9 180-199 129 129 129 225 225 225 F F F 10 200-219 129 129 129 225 225 225 F F F 11 220-239 129 129 129 225 225 225 F F F 12 240-259 124 124 124 219 219 219 F F F 13 260-279 121 121 121 210 210 210 F F F 14 280-299 128 128 128 224 224 224 F F F 15 300-319 128 128 128 224 224 224 F F F 16 320-339 129 129 129 225 225 225 F F F 17 340-359 129 129 129 225 225 225 F F F 18 360-379 126 126 126 221 221 221 F F F 19 380-399 129 129 129 225 225 225 F F F 20 400-419 122 122 122 214 214 214 F F F 21 420-439 127 127 127 223 223 223 F F F 22 440-459 127 127 127 223 223 223 F F F 23 460-479 129 129 129 225 225 225 F F F 24 480-499 129 129 129 225 225 225 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 124 124 124 216 216 216 F F F 27 540-559 127 127 127 223 223 223 F F F 28 560-579 129 129 129 225 225 225 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 129, Max flush: 225 Begin loop 65 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.401116261Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:39.445222064Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.445609685Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.477169049Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.477731468Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.477779222Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.47779495Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.477821667Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723} {"msg":"Crucible e62796e8-fc3f-4e28-89ec-c5c57dea7b7a has session id: a4f0a0dd-ffaf-49f4-9d47-03dd0020183a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.478378237Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"Upstairs opts: Upstairs UUID: e62796e8-fc3f-4e28-89ec-c5c57dea7b7a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.478438909Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"Crucible stats registered with UUID: e62796e8-fc3f-4e28-89ec-c5c57dea7b7a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.478464236Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:39.478484573Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.478690487Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"io task","client":"0","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.478730022Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"io task","client":"1","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.47874995Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"io task","client":"2","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.478879683Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"task":"crutest"} {"msg":"The guest has requested activation with gen:130","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.478900911Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723} {"msg":"e62796e8-fc3f-4e28-89ec-c5c57dea7b7a active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.478960273Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.478993579Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"client":"0","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.479015117Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"client":"1","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.479041363Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"client":"2","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.479060931Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"io task","client":"0","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.479092397Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"io task","client":"2","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.479126952Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"io task","client":"1","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"ds_connection connected from Ok(127.0.0.1:51899)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.479170887Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"io task","client":"0","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"ds_connection connected from Ok(127.0.0.1:46941)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.479395378Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"io task","client":"1","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"ds_connection connected from Ok(127.0.0.1:58454)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.479443532Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"io task","client":"2","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.480398172Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"client":"1","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.480431998Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"client":"0","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.480452435Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"client":"2","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.48097343Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481002836Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"[0]R flush_numbers[0..12]: [220, 218, 225, 224, 225, 225, 225, 224, 225, 225, 225, 225]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481021624Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"downstairs","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"[0]R generation[0..12]: [125, 124, 129, 128, 129, 129, 129, 128, 129, 129, 129, 129]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481044501Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"downstairs","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481064968Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"downstairs","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"[1]R flush_numbers[0..12]: [220, 218, 225, 224, 225, 225, 225, 224, 225, 225, 225, 225]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481084196Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"downstairs","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"[1]R generation[0..12]: [125, 124, 129, 128, 129, 129, 129, 128, 129, 129, 129, 129]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481103043Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"downstairs","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481122981Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"downstairs","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"[2]R flush_numbers[0..12]: [220, 218, 225, 224, 225, 225, 225, 224, 225, 225, 225, 225]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481142009Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"downstairs","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"[2]R generation[0..12]: [125, 124, 129, 128, 129, 129, 129, 128, 129, 129, 129, 129]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481160976Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"downstairs","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481180364Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"downstairs","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"Max found gen is 130","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481196742Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"downstairs","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"Generation requested: 130 >= found:130","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481214589Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"downstairs","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"Next flush: 226","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481230797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"downstairs","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481246845Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"":"downstairs","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481263643Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481278781Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"client":"0","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481297759Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"client":"1","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-11-06T20:16:39.481314487Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"client":"2","":"downstairs","session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481336924Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481352972Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"e62796e8-fc3f-4e28-89ec-c5c57dea7b7a is now active with session: a4f0a0dd-ffaf-49f4-9d47-03dd0020183a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.48137089Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481386308Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723,"session_id":"a4f0a0dd-ffaf-49f4-9d47-03dd0020183a"} {"msg":"The guest has finished waiting for activation with:130","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481402336Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723} {"msg":"Activated sub_volume e62796e8-fc3f-4e28-89ec-c5c57dea7b7a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:39.481416204Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 306 len 16384 data: 9 11 10 11 03/30 Read block 444 len 28672 04/30 Write block 456 len 20480 data: 16 15 13 12 7 05/30 Write block 479 len 36864 data: 10 11 12 11 12 12 10 6 6 06/30 Flush 07/30 Write block 453 len 12288 data: 14 16 16 08/30 Read block 571 len 8192 09/30 Read block 147 len 4096 10/30 Write block 551 len 12288 data: 11 11 12 11/30 Read block 305 len 24576 12/30 Write block 557 len 4096 data: 9 13/30 Write block 265 len 4096 data: 7 14/30 Read block 595 len 8192 15/30 Write block 282 len 32768 data: 8 8 7 7 7 7 6 13 16/30 Read block 254 len 24576 17/30 Read block 172 len 4096 18/30 Read block 532 len 20480 19/30 Write block 537 len 4096 data: 12 20/30 Write block 445 len 36864 data: 14 9 9 12 16 19 18 14 15 21/30 Write block 564 len 28672 data: 10 8 8 8 6 5 6 22/30 Read block 150 len 28672 23/30 Read block 74 len 24576 24/30 Read block 230 len 36864 25/30 Read block 390 len 24576 26/30 Read block 321 len 28672 27/30 Write block 530 len 20480 data: 7 11 12 12 11 28/30 Write block 347 len 36864 data: 8 8 8 10 8 8 10 9 9 29/30 Read block 96 len 24576 30/30 Write block 568 len 28672 data: 7 6 7 7 6 7 6 ---------------------------------------------------------------- Crucible gen:130 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 4 Done Done Done false 1002 Acked Read 7 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Write 9 Done Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 3 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Read 1 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Write 1 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Write 8 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Read 1 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Write 9 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Read 9 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Write 9 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 1 1 27 Done 4 28 28 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1005)) Some(JobId(1005)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:39.500503839Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1723} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 220 220 220 F F F 1 020-039 124 124 124 218 218 218 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 128 128 128 224 224 224 F F F 4 080-099 129 129 129 225 225 225 F F F 5 100-119 129 129 129 225 225 225 F F F 6 120-139 129 129 129 225 225 225 F F F 7 140-159 128 128 128 224 224 224 F F F 8 160-179 129 129 129 225 225 225 F F F 9 180-199 129 129 129 225 225 225 F F F 10 200-219 129 129 129 225 225 225 F F F 11 220-239 129 129 129 225 225 225 F F F 12 240-259 124 124 124 219 219 219 F F F 13 260-279 121 121 121 210 210 210 F T T <--- 14 280-299 128 128 128 224 224 224 F T T <--- 15 300-319 128 130 130 224 227 227 T F F <--- 16 320-339 129 129 129 225 225 225 F F F 17 340-359 129 129 129 225 225 225 F T T <--- 18 360-379 126 126 126 221 221 221 F F F 19 380-399 129 129 129 225 225 225 F F F 20 400-419 122 122 122 214 214 214 F F F 21 420-439 127 127 127 223 223 223 F F F 22 440-459 127 130 130 223 227 227 T T T <--- 23 460-479 129 130 130 225 227 227 T F F <--- 24 480-499 129 130 130 225 227 227 T F F <--- 25 500-519 116 116 116 203 203 203 F F F 26 520-539 124 124 124 216 216 216 F T T <--- 27 540-559 127 127 127 223 223 223 F T T <--- 28 560-579 129 129 129 225 225 225 F T T <--- 29 580-599 123 123 123 215 215 215 F F F Max gen: 130, Max flush: 227 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 65 Verifying data now No change after verify Loop: 65 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 220 220 220 F F F 1 020-039 124 124 124 218 218 218 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 128 128 128 224 224 224 F F F 4 080-099 129 129 129 225 225 225 F F F 5 100-119 129 129 129 225 225 225 F F F 6 120-139 129 129 129 225 225 225 F F F 7 140-159 128 128 128 224 224 224 F F F 8 160-179 129 129 129 225 225 225 F F F 9 180-199 129 129 129 225 225 225 F F F 10 200-219 129 129 129 225 225 225 F F F 11 220-239 129 129 129 225 225 225 F F F 12 240-259 124 124 124 219 219 219 F F F 13 260-279 131 131 131 228 228 228 F F F 14 280-299 131 131 131 228 228 228 F F F 15 300-319 130 130 130 227 227 227 F F F 16 320-339 129 129 129 225 225 225 F F F 17 340-359 131 131 131 228 228 228 F F F 18 360-379 126 126 126 221 221 221 F F F 19 380-399 129 129 129 225 225 225 F F F 20 400-419 122 122 122 214 214 214 F F F 21 420-439 127 127 127 223 223 223 F F F 22 440-459 131 131 131 228 228 228 F F F 23 460-479 130 130 130 227 227 227 F F F 24 480-499 130 130 130 227 227 227 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 131 131 131 228 228 228 F F F 27 540-559 131 131 131 228 228 228 F F F 28 560-579 131 131 131 228 228 228 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 131, Max flush: 228 Begin loop 66 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:43.933656435Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:43.977779696Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:43.978183335Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.009709413Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.010380718Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.010428082Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.01044382Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.010470517Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731} {"msg":"Crucible 54edd0d4-d1fc-4be7-897d-24878c825124 has session id: e16625e8-92fb-486c-a44a-492719ee95ac","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.010932289Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"Upstairs opts: Upstairs UUID: 54edd0d4-d1fc-4be7-897d-24878c825124, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.010986152Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"Crucible stats registered with UUID: 54edd0d4-d1fc-4be7-897d-24878c825124","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011013369Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:44.011031116Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011227592Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"io task","client":"0","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011262767Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"io task","client":"2","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011281375Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"io task","client":"1","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011406329Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"task":"crutest"} {"msg":"The guest has requested activation with gen:132","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011427807Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731} {"msg":"54edd0d4-d1fc-4be7-897d-24878c825124 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011497448Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011516975Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"client":"0","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011533983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"client":"1","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011550771Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"client":"2","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011576798Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"io task","client":"2","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011603464Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"io task","client":"0","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011625051Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"io task","client":"1","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:41907)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011706791Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"io task","client":"2","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:38329)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011748906Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"io task","client":"1","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"ds_connection connected from Ok(127.0.0.1:52599)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.011781662Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"io task","client":"0","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.012942545Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"client":"1","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.012976631Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"client":"2","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.013265745Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"client":"0","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.013486517Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.013561058Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"[0]R flush_numbers[0..12]: [220, 218, 225, 224, 225, 225, 225, 224, 225, 225, 225, 225]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.013793638Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"downstairs","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"[0]R generation[0..12]: [125, 124, 129, 128, 129, 129, 129, 128, 129, 129, 129, 129]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.013817605Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"downstairs","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.013837813Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"downstairs","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"[1]R flush_numbers[0..12]: [220, 218, 225, 224, 225, 225, 225, 224, 225, 225, 225, 225]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.01385706Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"downstairs","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"[1]R generation[0..12]: [125, 124, 129, 128, 129, 129, 129, 128, 129, 129, 129, 129]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.013876298Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"downstairs","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.013895965Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"downstairs","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"[2]R flush_numbers[0..12]: [220, 218, 225, 224, 225, 225, 225, 224, 225, 225, 225, 225]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.013918853Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"downstairs","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"[2]R generation[0..12]: [125, 124, 129, 128, 129, 129, 129, 128, 129, 129, 129, 129]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.01393835Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"downstairs","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.013958178Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"downstairs","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"Max found gen is 132","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.013974686Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"downstairs","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"Generation requested: 132 >= found:132","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.013991633Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"downstairs","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"Next flush: 229","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.014007831Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"downstairs","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.014023909Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"":"downstairs","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.014040897Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.014056085Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"client":"0","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.014073373Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"client":"1","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2025-11-06T20:16:44.01409634Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"client":"2","":"downstairs","session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.014113898Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.014130076Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"54edd0d4-d1fc-4be7-897d-24878c825124 is now active with session: e16625e8-92fb-486c-a44a-492719ee95ac","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.014147904Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.014163052Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731,"session_id":"e16625e8-92fb-486c-a44a-492719ee95ac"} {"msg":"The guest has finished waiting for activation with:132","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.014181169Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731} {"msg":"Activated sub_volume 54edd0d4-d1fc-4be7-897d-24878c825124","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:44.014195178Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 333 len 32768 data: 12 12 14 13 14 13 12 10 02/30 Flush 03/30 Write block 342 len 12288 data: 7 7 7 04/30 Write block 358 len 28672 data: 8 8 6 8 6 8 8 05/30 Write block 164 len 36864 data: 10 8 9 9 10 10 7 7 6 06/30 Read block 523 len 20480 07/30 Read block 187 len 16384 08/30 Read block 573 len 8192 09/30 Read block 344 len 36864 10/30 Read block 363 len 16384 11/30 Read block 472 len 12288 12/30 Read block 339 len 40960 13/30 Write block 108 len 40960 data: 9 10 12 14 11 12 15 13 20 17 14/30 Write block 464 len 20480 data: 10 11 10 11 9 15/30 Read block 60 len 40960 16/30 Read block 353 len 8192 17/30 Read block 495 len 36864 18/30 Flush 19/30 Write block 177 len 4096 data: 10 20/30 Write block 174 len 28672 data: 8 7 9 11 12 17 19 21/30 Write block 289 len 16384 data: 14 15 13 13 22/30 Read block 579 len 4096 23/30 Flush 24/30 Write block 353 len 16384 data: 11 10 10 9 25/30 Write block 275 len 12288 data: 12 10 11 26/30 Flush 27/30 Write block 132 len 16384 data: 15 15 14 11 28/30 Read block 160 len 20480 29/30 Write block 54 len 40960 data: 9 7 6 7 9 7 7 5 4 4 30/30 Read block 148 len 4096 ---------------------------------------------------------------- Crucible gen:132 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Read 5 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Read 2 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Write 10 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Read 2 Sent Done Done false 1016 Acked Read 9 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Write 4 Sent Done Done false 1021 Acked Read 1 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Write 10 Sent Done Done false 1029 Acked Read 1 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:44.03338568Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1731} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 220 220 220 F F F 1 020-039 124 124 124 218 218 218 F F F 2 040-059 129 129 129 225 225 225 F T T <--- 3 060-079 128 128 128 224 224 224 F T T <--- 4 080-099 129 129 129 225 225 225 F F F 5 100-119 129 132 132 225 230 230 F F F <--- 6 120-139 129 129 129 225 225 225 F T T <--- 7 140-159 128 128 128 224 224 224 F F F 8 160-179 129 132 132 225 231 231 F F F <--- 9 180-199 129 132 132 225 231 231 F F F <--- 10 200-219 129 129 129 225 225 225 F F F 11 220-239 129 129 129 225 225 225 F F F 12 240-259 124 124 124 219 219 219 F F F 13 260-279 131 132 132 228 232 232 F F F <--- 14 280-299 131 132 132 228 231 231 F F F <--- 15 300-319 130 130 130 227 227 227 F F F 16 320-339 129 132 132 225 229 229 F F F <--- 17 340-359 131 132 132 228 232 232 F F F <--- 18 360-379 126 132 132 221 230 230 F F F <--- 19 380-399 129 129 129 225 225 225 F F F 20 400-419 122 122 122 214 214 214 F F F 21 420-439 127 127 127 223 223 223 F F F 22 440-459 131 131 131 228 228 228 F F F 23 460-479 130 132 132 227 230 230 F F F <--- 24 480-499 130 130 130 227 227 227 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 131 131 131 228 228 228 F F F 27 540-559 131 131 131 228 228 228 F F F 28 560-579 131 131 131 228 228 228 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 132, Max flush: 232 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 66 Verifying data now No change after verify Loop: 66 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 220 220 220 F F F 1 020-039 124 124 124 218 218 218 F F F 2 040-059 133 133 133 233 233 233 F F F 3 060-079 133 133 133 233 233 233 F F F 4 080-099 129 129 129 225 225 225 F F F 5 100-119 132 132 132 230 230 230 F F F 6 120-139 133 133 133 233 233 233 F F F 7 140-159 128 128 128 224 224 224 F F F 8 160-179 132 132 132 231 231 231 F F F 9 180-199 132 132 132 231 231 231 F F F 10 200-219 129 129 129 225 225 225 F F F 11 220-239 129 129 129 225 225 225 F F F 12 240-259 124 124 124 219 219 219 F F F 13 260-279 132 132 132 232 232 232 F F F 14 280-299 132 132 132 231 231 231 F F F 15 300-319 130 130 130 227 227 227 F F F 16 320-339 132 132 132 229 229 229 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 129 129 129 225 225 225 F F F 20 400-419 122 122 122 214 214 214 F F F 21 420-439 127 127 127 223 223 223 F F F 22 440-459 131 131 131 228 228 228 F F F 23 460-479 132 132 132 230 230 230 F F F 24 480-499 130 130 130 227 227 227 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 131 131 131 228 228 228 F F F 27 540-559 131 131 131 228 228 228 F F F 28 560-579 131 131 131 228 228 228 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 133, Max flush: 233 Begin loop 67 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.462604961Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.511422491Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.512038853Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.512087627Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.512105395Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.512127752Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739} {"msg":"Crucible 11a08ac1-57c2-4836-80c4-2cc09d3f55ef has session id: 34942c01-20f9-4c96-b704-6723692c1ecd","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.512607272Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"Upstairs opts: Upstairs UUID: 11a08ac1-57c2-4836-80c4-2cc09d3f55ef, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.512686162Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"Crucible stats registered with UUID: 11a08ac1-57c2-4836-80c4-2cc09d3f55ef","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.512718818Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:48.512741715Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.512923352Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"io task","client":"0","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.512956528Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"io task","client":"2","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.512979905Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"io task","client":"1","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.513138155Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"task":"crutest"} {"msg":"The guest has requested activation with gen:134","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.513161212Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739} {"msg":"11a08ac1-57c2-4836-80c4-2cc09d3f55ef active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.513230473Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.513250021Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"client":"0","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.513267339Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"client":"1","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.513283987Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"client":"2","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.513300505Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"io task","client":"2","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.513327131Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"io task","client":"0","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.513371516Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"io task","client":"1","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"ds_connection connected from Ok(127.0.0.1:58348)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.513476582Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"io task","client":"2","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"ds_connection connected from Ok(127.0.0.1:61937)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.513518797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"io task","client":"1","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"ds_connection connected from Ok(127.0.0.1:59757)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.513551913Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"io task","client":"0","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.51483964Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"client":"0","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.514891254Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"client":"2","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.515512016Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.515546062Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.515892528Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"client":"1","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"[0]R flush_numbers[0..12]: [220, 218, 233, 233, 225, 230, 233, 224, 231, 231, 225, 225]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516410593Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"downstairs","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"[0]R generation[0..12]: [125, 124, 133, 133, 129, 132, 133, 128, 132, 132, 129, 129]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.51643567Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"downstairs","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516456287Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"downstairs","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"[1]R flush_numbers[0..12]: [220, 218, 233, 233, 225, 230, 233, 224, 231, 231, 225, 225]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516475585Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"downstairs","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"[1]R generation[0..12]: [125, 124, 133, 133, 129, 132, 133, 128, 132, 132, 129, 129]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516505081Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"downstairs","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516525938Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"downstairs","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"[2]R flush_numbers[0..12]: [220, 218, 233, 233, 225, 230, 233, 224, 231, 231, 225, 225]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516544916Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"downstairs","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"[2]R generation[0..12]: [125, 124, 133, 133, 129, 132, 133, 128, 132, 132, 129, 129]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516563463Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"downstairs","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516582671Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"downstairs","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"Max found gen is 134","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516599029Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"downstairs","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"Generation requested: 134 >= found:134","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516616137Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"downstairs","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"Next flush: 234","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516632025Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"downstairs","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516652222Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"":"downstairs","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} Wait for a query_work_queue command to finish before sending IO {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516677019Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516692367Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"client":"0","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516709305Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"client":"1","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516726143Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"client":"2","":"downstairs","session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516743161Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516759219Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"11a08ac1-57c2-4836-80c4-2cc09d3f55ef is now active with session: 34942c01-20f9-4c96-b704-6723692c1ecd","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516777196Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516792225Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739,"session_id":"34942c01-20f9-4c96-b704-6723692c1ecd"} {"msg":"The guest has finished waiting for activation with:134","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516808152Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739} {"msg":"Activated sub_volume 11a08ac1-57c2-4836-80c4-2cc09d3f55ef","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:16:48.516821951Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 74 len 20480 data: 6 7 9 10 14 02/30 Write block 422 len 8192 data: 13 11 03/30 Write block 132 len 16384 data: 16 16 15 12 04/30 Read block 163 len 32768 05/30 Read block 149 len 4096 06/30 Write block 89 len 20480 data: 12 11 11 8 7 07/30 Read block 197 len 16384 08/30 Read block 16 len 36864 09/30 Read block 44 len 32768 10/30 Write block 162 len 32768 data: 12 10 11 9 10 10 11 11 11/30 Read block 481 len 20480 12/30 Read block 500 len 12288 13/30 Write block 459 len 28672 data: 13 8 7 8 8 11 12 14/30 Write block 415 len 24576 data: 8 10 11 15 14 14 15/30 Write block 400 len 24576 data: 9 11 10 10 11 11 16/30 Write block 292 len 40960 data: 14 14 13 12 10 8 10 10 7 8 17/30 Read block 598 len 8192 18/30 Read block 334 len 12288 19/30 Flush 20/30 Write block 10 len 32768 data: 9 11 9 7 6 4 3 3 21/30 Write block 98 len 24576 data: 9 10 8 8 9 7 22/30 Write block 428 len 40960 data: 11 10 11 9 7 7 11 10 15 15 23/30 Write block 92 len 8192 data: 9 8 24/30 Write block 470 len 36864 data: 11 12 14 14 12 13 11 10 12 25/30 Write block 157 len 16384 data: 9 9 11 11 26/30 Write block 38 len 40960 data: 22 19 21 21 18 15 15 16 15 15 27/30 Write block 106 len 24576 data: 9 10 10 11 13 15 28/30 Read block 17 len 28672 29/30 Read block 89 len 4096 30/30 Read block 572 len 24576 ---------------------------------------------------------------- Crucible gen:134 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Read 1 Done Done Done false 1005 Acked Write 5 Done Done Done false 1006 Acked Read 4 Done Done Done false 1007 Acked Read 9 Done Done Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Write 8 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Read 3 Done Sent Done false 1012 Acked Write 7 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Write 6 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Read 3 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 8 Done Sent Done false 1020 Acked Write 6 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Write 2 Done Sent Done false 1023 Acked Write 9 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Read 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 22 0 22 Done 30 8 30 68 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) None Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:16:48.53548268Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1739} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 125 125 125 220 220 220 T F T <--- 1 020-039 124 124 124 218 218 218 T F T <--- 2 040-059 133 133 133 233 233 233 T F T <--- 3 060-079 134 133 134 234 233 234 F T F <--- 4 080-099 134 129 134 234 225 234 T T T <--- 5 100-119 132 132 132 230 230 230 T F T <--- 6 120-139 134 133 134 234 233 234 F T F <--- 7 140-159 128 128 128 224 224 224 T F T <--- 8 160-179 134 132 134 234 231 234 T F T <--- 9 180-199 132 132 132 231 231 231 F F F 10 200-219 129 129 129 225 225 225 F F F 11 220-239 129 129 129 225 225 225 F F F 12 240-259 124 124 124 219 219 219 F F F 13 260-279 132 132 132 232 232 232 F F F 14 280-299 134 132 134 234 231 234 F F F <--- 15 300-319 134 130 134 234 227 234 F F F <--- 16 320-339 132 132 132 229 229 229 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 129 129 129 225 225 225 F F F 20 400-419 134 122 134 234 214 234 F F F <--- 21 420-439 134 127 134 234 223 234 T T T <--- 22 440-459 134 131 134 234 228 234 F F F <--- 23 460-479 134 132 134 234 230 234 T F T <--- 24 480-499 130 130 130 227 227 227 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 131 131 131 228 228 228 F F F 27 540-559 131 131 131 228 228 228 F F F 28 560-579 131 131 131 228 228 228 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 134, Max flush: 234 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 67 Verifying data now No change after verify Loop: 67 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 235 235 235 F F F 1 020-039 135 135 135 235 235 235 F F F 2 040-059 135 135 135 235 235 235 F F F 3 060-079 134 134 134 234 234 234 F F F 4 080-099 135 135 135 235 235 235 F F F 5 100-119 135 135 135 235 235 235 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 135 135 135 235 235 235 F F F 8 160-179 135 135 135 235 235 235 F F F 9 180-199 132 132 132 231 231 231 F F F 10 200-219 129 129 129 225 225 225 F F F 11 220-239 129 129 129 225 225 225 F F F 12 240-259 124 124 124 219 219 219 F F F 13 260-279 132 132 132 232 232 232 F F F 14 280-299 134 134 134 234 234 234 F F F 15 300-319 134 134 134 234 234 234 F F F 16 320-339 132 132 132 229 229 229 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 129 129 129 225 225 225 F F F 20 400-419 134 134 134 234 234 234 F F F 21 420-439 135 135 135 235 235 235 F F F 22 440-459 134 134 134 234 234 234 F F F 23 460-479 135 135 135 235 235 235 F F F 24 480-499 130 130 130 227 227 227 F F F 25 500-519 116 116 116 203 203 203 F F F 26 520-539 131 131 131 228 228 228 F F F 27 540-559 131 131 131 228 228 228 F F F 28 560-579 131 131 131 228 228 228 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 135, Max flush: 235 Begin loop 68 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:02.955693986Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.000900651Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.001511934Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.001558848Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.001578955Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.001601622Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747} {"msg":"Crucible d5c74da4-02d4-4722-a576-213102e17e46 has session id: 3c8da68c-9c12-4069-957e-9293e80e01a4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.002238142Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"Upstairs opts: Upstairs UUID: d5c74da4-02d4-4722-a576-213102e17e46, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.002295675Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"Crucible stats registered with UUID: d5c74da4-02d4-4722-a576-213102e17e46","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.002326341Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:03.002352148Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.002534545Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"0","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.00257555Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"2","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.002597847Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.002754657Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"task":"crutest"} {"msg":"The guest has requested activation with gen:136","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.002775505Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747} {"msg":"d5c74da4-02d4-4722-a576-213102e17e46 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.002863483Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.002884191Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"client":"0","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.002905188Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"client":"1","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.002934405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"client":"2","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.00297142Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"2","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.003004926Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"0","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.003035632Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"ds_connection connected from Ok(127.0.0.1:60935)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.003064158Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"2","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"ds_connection connected from Ok(127.0.0.1:43757)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.003093094Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"0","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:03.003133589Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:03.003173364Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:03.00321209Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.003384078Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.003417054Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.00432327Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"client":"2","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.004441935Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"client":"0","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.004808189Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:03.004995715Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.004965499Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"ds_connection connected from Ok(127.0.0.1:46532)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.005179122Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"io task","client":"1","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.006488797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"client":"1","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"[0]R flush_numbers[0..12]: [235, 235, 235, 234, 235, 235, 234, 235, 235, 231, 225, 225]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007166472Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"downstairs","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"[0]R generation[0..12]: [135, 135, 135, 134, 135, 135, 134, 135, 135, 132, 129, 129]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007202557Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"downstairs","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007228434Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"downstairs","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"[1]R flush_numbers[0..12]: [235, 235, 235, 234, 235, 235, 234, 235, 235, 231, 225, 225]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007252631Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"downstairs","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"[1]R generation[0..12]: [135, 135, 135, 134, 135, 135, 134, 135, 135, 132, 129, 129]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007276568Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"downstairs","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007301505Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"downstairs","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"[2]R flush_numbers[0..12]: [235, 235, 235, 234, 235, 235, 234, 235, 235, 231, 225, 225]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007325042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"downstairs","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"[2]R generation[0..12]: [135, 135, 135, 134, 135, 135, 134, 135, 135, 132, 129, 129]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007348409Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"downstairs","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007372836Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"downstairs","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"Max found gen is 136","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007393193Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"downstairs","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"Generation requested: 136 >= found:136","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.00741469Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"downstairs","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"Next flush: 236","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007438818Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"downstairs","":"downstairs"Wait for a query_work_queue command to finish before sending IO ,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007467304Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"":"downstairs","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007488891Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007508209Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"client":"0","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007529546Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"client":"1","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007550723Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"client":"2","":"downstairs","session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007571991Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007592298Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"d5c74da4-02d4-4722-a576-213102e17e46 is now active with session: 3c8da68c-9c12-4069-957e-9293e80e01a4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007614815Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007633633Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747,"session_id":"3c8da68c-9c12-4069-957e-9293e80e01a4"} {"msg":"The guest has finished waiting for activation with:136","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.0076539Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747} {"msg":"Activated sub_volume d5c74da4-02d4-4722-a576-213102e17e46","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:13.007671558Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 91 len 36864 data: 12 10 9 9 10 10 11 10 11 02/30 Read block 86 len 36864 03/30 Write block 22 len 32768 data: 14 15 15 16 15 13 12 10 04/30 Write block 124 len 4096 data: 11 05/30 Read block 341 len 24576 06/30 Write block 132 len 16384 data: 17 17 16 13 07/30 Write block 487 len 24576 data: 7 7 6 4 4 7 08/30 Write block 189 len 40960 data: 10 8 10 10 9 9 8 10 14 13 09/30 Read block 261 len 40960 10/30 Read block 64 len 20480 11/30 Flush 12/30 Write block 119 len 32768 data: 16 17 11 10 12 12 11 10 13/30 Flush 14/30 Write block 237 len 40960 data: 9 11 12 11 12 15 17 17 16 15 15/30 Read block 537 len 40960 16/30 Write block 518 len 36864 data: 10 9 9 9 7 6 8 10 9 17/30 Read block 238 len 24576 18/30 Read block 175 len 28672 19/30 Write block 436 len 4096 data: 16 20/30 Read block 407 len 4096 21/30 Read block 512 len 4096 22/30 Flush 23/30 Write block 110 len 24576 data: 14 16 12 13 16 14 24/30 Read block 194 len 16384 25/30 Flush 26/30 Write block 529 len 20480 data: 8 8 12 13 13 27/30 Write block 463 len 8192 data: 9 12 28/30 Write block 478 len 4096 data: 13 29/30 Read block 388 len 32768 30/30 Read block 541 len 24576 ---------------------------------------------------------------- Crucible gen:136 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Write 1 Done Sent Done false 1004 Acked Read 6 Done Sent Done false 1005 Acked Write 4 Done Sent Done false 1006 Acked Write 6 Done Sent Done false 1007 Acked Write 10 Done Sent Done false 1008 Acked Read 10 Done Sent Done false 1009 Acked Read 5 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 8 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 10 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Read 6 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Write 1 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Read 1 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Flush 0 Done Sent Done false 1025 Acked Write 5 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Read 8 Done Sent Done false 1029 Acked Read 6 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 27 0 27 Done 30 3 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) None Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:13.028118882Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1747} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 235 235 235 F F F 1 020-039 136 135 136 236 235 236 F T F <--- 2 040-059 135 135 135 235 235 235 F F F 3 060-079 134 134 134 234 234 234 F F F 4 080-099 136 135 136 236 235 236 F T F <--- 5 100-119 136 135 136 239 235 239 F F F <--- 6 120-139 136 134 136 237 234 237 F F F <--- 7 140-159 135 135 135 235 235 235 F F F 8 160-179 135 135 135 235 235 235 F F F 9 180-199 136 132 136 236 231 236 F F F <--- 10 200-219 129 129 129 225 225 225 F F F 11 220-239 136 129 136 238 225 238 F F F <--- 12 240-259 136 124 136 238 219 238 F F F <--- 13 260-279 132 132 132 232 232 232 F F F 14 280-299 134 134 134 234 234 234 F F F 15 300-319 134 134 134 234 234 234 F F F 16 320-339 132 132 132 229 229 229 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 129 129 129 225 225 225 F F F 20 400-419 134 134 134 234 234 234 F F F 21 420-439 136 135 136 238 235 238 F F F <--- 22 440-459 134 134 134 234 234 234 F F F 23 460-479 135 135 135 235 235 235 T F T <--- 24 480-499 136 130 136 236 227 236 F F F <--- 25 500-519 136 116 136 238 203 238 F F F <--- 26 520-539 136 131 136 238 228 238 T F T <--- 27 540-559 131 131 131 228 228 228 F F F 28 560-579 131 131 131 228 228 228 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 136, Max flush: 239 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 68 Verifying data now No change after verify Loop: 68 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 235 235 235 F F F 1 020-039 136 136 136 236 236 236 F F F 2 040-059 135 135 135 235 235 235 F F F 3 060-079 134 134 134 234 234 234 F F F 4 080-099 136 136 136 236 236 236 F F F 5 100-119 136 136 136 239 239 239 F F F 6 120-139 136 136 136 237 237 237 F F F 7 140-159 135 135 135 235 235 235 F F F 8 160-179 135 135 135 235 235 235 F F F 9 180-199 136 136 136 236 236 236 F F F 10 200-219 129 129 129 225 225 225 F F F 11 220-239 136 136 136 238 238 238 F F F 12 240-259 136 136 136 238 238 238 F F F 13 260-279 132 132 132 232 232 232 F F F 14 280-299 134 134 134 234 234 234 F F F 15 300-319 134 134 134 234 234 234 F F F 16 320-339 132 132 132 229 229 229 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 129 129 129 225 225 225 F F F 20 400-419 134 134 134 234 234 234 F F F 21 420-439 136 136 136 238 238 238 F F F 22 440-459 134 134 134 234 234 234 F F F 23 460-479 137 137 137 240 240 240 F F F 24 480-499 136 136 136 236 236 236 F F F 25 500-519 136 136 136 238 238 238 F F F 26 520-539 137 137 137 240 240 240 F F F 27 540-559 131 131 131 228 228 228 F F F 28 560-579 131 131 131 228 228 228 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 137, Max flush: 240 Begin loop 69 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.43004993Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:17.473969987Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.474366167Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.506227703Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.506764865Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.506817958Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.506834227Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.506855924Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755} {"msg":"Crucible ba293f74-c3db-449a-ad68-99a452445ad4 has session id: 7ec861cc-4ef2-40a3-91ce-473417a21511","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.507430681Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"Upstairs opts: Upstairs UUID: ba293f74-c3db-449a-ad68-99a452445ad4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.507471986Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"Crucible stats registered with UUID: ba293f74-c3db-449a-ad68-99a452445ad4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.507491544Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:17.507508402Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.507737033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"io task","client":"0","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.507776128Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"io task","client":"2","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.507796685Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"io task","client":"1","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.507940287Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"task":"crutest"} {"msg":"The guest has requested activation with gen:138","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.507961374Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755} {"msg":"ba293f74-c3db-449a-ad68-99a452445ad4 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.508046044Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.508065751Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"client":"0","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.508088049Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"client":"1","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.508105436Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"client":"2","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.508122034Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"io task","client":"2","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.508139142Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"io task","client":"0","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.50815635Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"io task","client":"1","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"ds_connection connected from Ok(127.0.0.1:48976)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.508267726Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"io task","client":"2","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"ds_connection connected from Ok(127.0.0.1:34756)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.50831185Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"io task","client":"0","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"ds_connection connected from Ok(127.0.0.1:40028)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.508345066Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"io task","client":"1","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.509495561Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"client":"1","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.509535146Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"client":"0","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.509668669Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"client":"2","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510044422Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510124522Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"[0]R flush_numbers[0..12]: [235, 236, 235, 234, 236, 239, 237, 235, 235, 236, 225, 238]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.5102232Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"downstairs","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"[0]R generation[0..12]: [135, 136, 135, 134, 136, 136, 136, 135, 135, 136, 129, 136]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510247856Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"downstairs","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510271223Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"downstairs","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"[1]R flush_numbers[0..12]: [235, 236, 235, 234, 236, 239, 237, 235, 235, 236, 225, 238]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510292231Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"downstairs","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"[1]R generation[0..12]: [135, 136, 135, 134, 136, 136, 136, 135, 135, 136, 129, 136]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510311408Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"downstairs","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510331076Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"downstairs","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"[2]R flush_numbers[0..12]: [235, 236, 235, 234, 236, 239, 237, 235, 235, 236, 225, 238]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510350204Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"downstairs","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"[2]R generation[0..12]: [135, 136, 135, 134, 136, 136, 136, 135, 135, 136, 129, 136]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510368971Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"downstairs","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510388839Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"downstairs","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"Max found gen is 138","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510405027Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"Generation requested: 138 >= found:138","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510427934Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"downstairs","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"Next flush: 241","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510444062Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"downstairs","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.51046023Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"":"downstairs","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510477397Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510492555Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"client":"0","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510509344Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"client":"1","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510526131Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"client":"2","":"downstairs","session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510544839Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510561177Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"ba293f74-c3db-449a-ad68-99a452445ad4 is now active with session: 7ec861cc-4ef2-40a3-91ce-473417a21511","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510579275Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:17:17.510594503Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755,"session_id":"7ec861cc-4ef2-40a3-91ce-473417a21511"} {"msg":"The guest has finished waiting for activation with:138","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510612791Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755} {"msg":"Activated sub_volume ba293f74-c3db-449a-ad68-99a452445ad4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:17.510627199Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 249 len 32768 data: 12 12 10 8 9 11 10 10 02/30 Read block 356 len 24576 03/30 Write block 324 len 20480 data: 11 12 12 11 11 04/30 Flush 05/30 Write block 237 len 36864 data: 10 12 13 12 13 16 18 18 17 06/30 Read block 516 len 32768 07/30 Read block 213 len 8192 08/30 Read block 110 len 24576 09/30 Write block 27 len 40960 data: 14 13 11 10 10 12 13 15 18 19 10/30 Write block 448 len 32768 data: 13 17 20 19 15 16 17 17 11/30 Write block 137 len 4096 data: 13 12/30 Read block 445 len 36864 13/30 Read block 50 len 16384 14/30 Read block 283 len 8192 15/30 Read block 53 len 4096 16/30 Write block 181 len 36864 data: 20 20 20 14 14 14 13 13 11 17/30 Read block 464 len 32768 18/30 Read block 353 len 20480 19/30 Write block 461 len 8192 data: 8 9 20/30 Read block 481 len 4096 21/30 Write block 46 len 36864 data: 16 16 13 12 12 10 10 9 10 22/30 Flush 23/30 Write block 277 len 32768 data: 12 11 10 8 8 9 9 8 24/30 Write block 287 len 4096 data: 8 25/30 Flush 26/30 Write block 540 len 40960 data: 15 15 19 15 13 12 9 10 11 10 27/30 Write block 125 len 16384 data: 12 11 12 13 28/30 Read block 359 len 20480 29/30 Write block 495 len 24576 data: 5 4 6 8 10 8 30/30 Write block 15 len 12288 data: 5 4 4 ---------------------------------------------------------------- Crucible gen:138 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Read 6 Sent Done Done false 1002 Acked Write 5 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Read 6 Sent Done Done false 1008 Acked Write 10 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Read 4 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Read 1 Sent Done Done false 1015 Acked Write 9 Sent Done Done false 1016 Acked Read 8 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:17.530121003Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1755} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 135 135 135 235 235 235 F T T <--- 1 020-039 136 138 138 236 242 242 F F F <--- 2 040-059 135 138 138 235 242 242 F F F <--- 3 060-079 134 134 134 234 234 234 F F F 4 080-099 136 136 136 236 236 236 F F F 5 100-119 136 136 136 239 239 239 F F F 6 120-139 136 138 138 237 242 242 F T T <--- 7 140-159 135 135 135 235 235 235 F F F 8 160-179 135 135 135 235 235 235 F F F 9 180-199 136 138 138 236 242 242 F F F <--- 10 200-219 129 129 129 225 225 225 F F F 11 220-239 136 138 138 238 242 242 F F F <--- 12 240-259 136 138 138 238 242 242 F F F <--- 13 260-279 132 138 138 232 243 243 F F F <--- 14 280-299 134 138 138 234 243 243 F F F <--- 15 300-319 134 134 134 234 234 234 F F F 16 320-339 132 138 138 229 241 241 F F F <--- 17 340-359 132 132 132 232 232 232 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 129 129 129 225 225 225 F F F 20 400-419 134 134 134 234 234 234 F F F 21 420-439 136 136 136 238 238 238 F F F 22 440-459 134 138 138 234 242 242 F F F <--- 23 460-479 137 138 138 240 242 242 F F F <--- 24 480-499 136 136 136 236 236 236 F T T <--- 25 500-519 136 136 136 238 238 238 F T T <--- 26 520-539 137 137 137 240 240 240 F F F 27 540-559 131 131 131 228 228 228 F T T <--- 28 560-579 131 131 131 228 228 228 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 138, Max flush: 243 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 69 Verifying data now No change after verify Loop: 69 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 139 139 139 244 244 244 F F F 1 020-039 138 138 138 242 242 242 F F F 2 040-059 138 138 138 242 242 242 F F F 3 060-079 134 134 134 234 234 234 F F F 4 080-099 136 136 136 236 236 236 F F F 5 100-119 136 136 136 239 239 239 F F F 6 120-139 139 139 139 244 244 244 F F F 7 140-159 135 135 135 235 235 235 F F F 8 160-179 135 135 135 235 235 235 F F F 9 180-199 138 138 138 242 242 242 F F F 10 200-219 129 129 129 225 225 225 F F F 11 220-239 138 138 138 242 242 242 F F F 12 240-259 138 138 138 242 242 242 F F F 13 260-279 138 138 138 243 243 243 F F F 14 280-299 138 138 138 243 243 243 F F F 15 300-319 134 134 134 234 234 234 F F F 16 320-339 138 138 138 241 241 241 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 129 129 129 225 225 225 F F F 20 400-419 134 134 134 234 234 234 F F F 21 420-439 136 136 136 238 238 238 F F F 22 440-459 138 138 138 242 242 242 F F F 23 460-479 138 138 138 242 242 242 F F F 24 480-499 139 139 139 244 244 244 F F F 25 500-519 139 139 139 244 244 244 F F F 26 520-539 137 137 137 240 240 240 F F F 27 540-559 139 139 139 244 244 244 F F F 28 560-579 131 131 131 228 228 228 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 139, Max flush: 244 Begin loop 70 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:21.975378872Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:22.019429023Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.019830542Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.051460497Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.052111245Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.052160329Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.052176347Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.052198844Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763} {"msg":"Crucible c77f894e-ccea-4251-a12e-9dcf99c2a5fb has session id: 2889920d-108a-4d74-ac42-427cfd492e53","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.052774062Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"Upstairs opts: Upstairs UUID: c77f894e-ccea-4251-a12e-9dcf99c2a5fb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.052818326Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"Crucible stats registered with UUID: c77f894e-ccea-4251-a12e-9dcf99c2a5fb","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.052838313Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:22.052854611Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053032879Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"io task","client":"0","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053071894Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"io task","client":"2","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053091062Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"io task","client":"1","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053304775Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"task":"crutest"} {"msg":"The guest has requested activation with gen:140","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.05334093Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763} {"msg":"c77f894e-ccea-4251-a12e-9dcf99c2a5fb active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053375886Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053392714Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"client":"0","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053410172Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"client":"1","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053427079Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"client":"2","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053443557Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"io task","client":"2","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053472784Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"io task","client":"0","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053564792Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"io task","client":"1","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"ds_connection connected from Ok(127.0.0.1:45227)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053627264Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"io task","client":"1","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"ds_connection connected from Ok(127.0.0.1:34942)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053667609Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"io task","client":"0","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"ds_connection connected from Ok(127.0.0.1:33698)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.053703065Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"io task","client":"2","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.054828483Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"client":"1","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.054864828Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"client":"0","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.054976524Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"client":"2","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055480521Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055509747Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"[0]R flush_numbers[0..12]: [244, 242, 242, 234, 236, 239, 244, 235, 235, 242, 225, 242]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055529234Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"downstairs","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"[0]R generation[0..12]: [139, 138, 138, 134, 136, 136, 139, 135, 135, 138, 129, 138]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055548932Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"downstairs","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055568889Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"downstairs","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"[1]R flush_numbers[0..12]: [244, 242, 242, 234, 236, 239, 244, 235, 235, 242, 225, 242]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055587937Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"downstairs","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"[1]R generation[0..12]: [139, 138, 138, 134, 136, 136, 139, 135, 135, 138, 129, 138]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055606865Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"downstairs","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055626512Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"downstairs","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"[2]R flush_numbers[0..12]: [244, 242, 242, 234, 236, 239, 244, 235, 235, 242, 225, 242]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055649859Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"downstairs","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"[2]R generation[0..12]: [139, 138, 138, 134, 136, 136, 139, 135, 135, 138, 129, 138]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055669327Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"downstairs","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055696584Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"downstairs","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"Max found gen is 140","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.05572321Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"downstairs","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"Generation requested: 140 >= found:140","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055740808Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"downstairs","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"Next flush: 245","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055756996Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"downstairs","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055773224Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"":"downstairs","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055790112Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.05580514Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"client":"0","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055822098Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"client":"1","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055841695Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"client":"2","":"downstairs","session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055863192Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.05588744Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"c77f894e-ccea-4251-a12e-9dcf99c2a5fb is now active with session: 2889920d-108a-4d74-ac42-427cfd492e53","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055919775Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055944282Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763,"session_id":"2889920d-108a-4d74-ac42-427cfd492e53"} {"msg":"The guest has finished waiting for activation with:140","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055973418Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763} {"msg":"Activated sub_volume c77f894e-ccea-4251-a12e-9dcf99c2a5fb","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:22.055997695Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 552 len 16384 data: 12 13 12 11 02/30 Flush 03/30 Write block 220 len 20480 data: 13 10 9 12 9 04/30 Flush 05/30 Write block 14 len 16384 data: 7 6 5 5 06/30 Read block 25 len 4096 07/30 Read block 527 len 28672 08/30 Write block 108 len 4096 data: 11 09/30 Flush 10/30 Write block 306 len 40960 data: 10 12 11 12 12 13 13 12 14 11 11/30 Read block 40 len 40960 12/30 Read block 301 len 32768 13/30 Flush 14/30 Write block 107 len 40960 data: 11 12 12 15 17 13 14 17 15 21 15/30 Write block 201 len 20480 data: 14 12 12 12 14 16/30 Read block 134 len 36864 17/30 Write block 121 len 28672 data: 12 11 13 13 13 12 13 18/30 Read block 59 len 24576 19/30 Read block 540 len 24576 20/30 Read block 419 len 24576 21/30 Write block 56 len 36864 data: 7 8 10 8 8 6 5 5 9 22/30 Read block 53 len 28672 23/30 Write block 207 len 32768 data: 9 6 6 7 6 6 7 7 24/30 Flush 25/30 Write block 184 len 24576 data: 15 15 15 14 14 12 26/30 Read block 108 len 40960 27/30 Read block 270 len 40960 28/30 Write block 537 len 32768 data: 13 13 14 16 16 20 16 14 29/30 Write block 530 len 16384 data: 9 13 14 14 30/30 Write block 306 len 16384 data: 11 13 12 13 ---------------------------------------------------------------- Crucible gen:140 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 5 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Read 1 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Read 10 Sent Done Done false 1011 Acked Read 8 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Read 6 Sent Done Done false 1018 Acked Read 6 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Read 7 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Write 8 Sent Sent Sent false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 3 3 35 Done 1 27 27 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:22.076319495Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1763} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 139 140 140 244 247 247 F F F <--- 1 020-039 138 138 138 242 242 242 F F F 2 040-059 138 140 140 242 249 249 F F F <--- 3 060-079 134 140 140 234 249 249 F F F <--- 4 080-099 136 136 136 236 236 236 F F F 5 100-119 136 140 140 239 249 249 F F F <--- 6 120-139 139 140 140 244 249 249 F F F <--- 7 140-159 135 135 135 235 235 235 F F F 8 160-179 135 135 135 235 235 235 F F F 9 180-199 138 138 138 242 242 242 F T T <--- 10 200-219 129 140 140 225 249 249 F F F <--- 11 220-239 138 140 140 242 246 246 F F F <--- 12 240-259 138 138 138 242 242 242 F F F 13 260-279 138 138 138 243 243 243 F F F 14 280-299 138 138 138 243 243 243 F F F 15 300-319 134 140 140 234 248 248 F T T <--- 16 320-339 138 138 138 241 241 241 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 129 129 129 225 225 225 F F F 20 400-419 134 134 134 234 234 234 F F F 21 420-439 136 136 136 238 238 238 F F F 22 440-459 138 138 138 242 242 242 F F F 23 460-479 138 138 138 242 242 242 F F F 24 480-499 139 139 139 244 244 244 F F F 25 500-519 139 139 139 244 244 244 F F F 26 520-539 137 137 137 240 240 240 F T T <--- 27 540-559 139 140 140 244 245 245 T T T <--- 28 560-579 131 131 131 228 228 228 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 140, Max flush: 249 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 70 Verifying data now No change after verify Loop: 70 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 247 247 247 F F F 1 020-039 138 138 138 242 242 242 F F F 2 040-059 140 140 140 249 249 249 F F F 3 060-079 140 140 140 249 249 249 F F F 4 080-099 136 136 136 236 236 236 F F F 5 100-119 140 140 140 249 249 249 F F F 6 120-139 140 140 140 249 249 249 F F F 7 140-159 135 135 135 235 235 235 F F F 8 160-179 135 135 135 235 235 235 F F F 9 180-199 141 141 141 250 250 250 F F F 10 200-219 140 140 140 249 249 249 F F F 11 220-239 140 140 140 246 246 246 F F F 12 240-259 138 138 138 242 242 242 F F F 13 260-279 138 138 138 243 243 243 F F F 14 280-299 138 138 138 243 243 243 F F F 15 300-319 141 141 141 250 250 250 F F F 16 320-339 138 138 138 241 241 241 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 129 129 129 225 225 225 F F F 20 400-419 134 134 134 234 234 234 F F F 21 420-439 136 136 136 238 238 238 F F F 22 440-459 138 138 138 242 242 242 F F F 23 460-479 138 138 138 242 242 242 F F F 24 480-499 139 139 139 244 244 244 F F F 25 500-519 139 139 139 244 244 244 F F F 26 520-539 141 141 141 250 250 250 F F F 27 540-559 141 141 141 250 250 250 F F F 28 560-579 131 131 131 228 228 228 F F F 29 580-599 123 123 123 215 215 215 F F F Max gen: 141, Max flush: 250 Begin loop 71 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.506374831Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:26.550383357Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.550783746Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.582181251Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.582720373Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.582767967Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.582784055Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.582805952Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771} {"msg":"Crucible 6a30ff3b-8357-4f43-b8a4-1c332cf2a2f2 has session id: d980a3f4-4596-4c20-9aa1-ab4a89c79613","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.58329558Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"Upstairs opts: Upstairs UUID: 6a30ff3b-8357-4f43-b8a4-1c332cf2a2f2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.583360372Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"Crucible stats registered with UUID: 6a30ff3b-8357-4f43-b8a4-1c332cf2a2f2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.583389329Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:26.583410406Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.58361476Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"io task","client":"0","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.583663194Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"io task","client":"2","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.583688411Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"io task","client":"1","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.583823594Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"task":"crutest"} {"msg":"The guest has requested activation with gen:142","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.58385108Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771} {"msg":"6a30ff3b-8357-4f43-b8a4-1c332cf2a2f2 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.583901654Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.583928641Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"client":"0","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.583946928Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"client":"1","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.583963936Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"client":"2","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.583980364Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"io task","client":"2","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.583997942Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"io task","client":"0","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.584019069Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"io task","client":"1","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"ds_connection connected from Ok(127.0.0.1:33381)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.584156152Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"io task","client":"2","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"ds_connection connected from Ok(127.0.0.1:52562)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.584203576Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"io task","client":"1","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"ds_connection connected from Ok(127.0.0.1:65132)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.584239871Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"io task","client":"0","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.585515591Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"client":"1","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.58559777Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"client":"2","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.585637145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"client":"0","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586137052Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586176237Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"[0]R flush_numbers[0..12]: [247, 242, 249, 249, 236, 249, 249, 235, 235, 250, 249, 246]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586206854Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"downstairs","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"[0]R generation[0..12]: [140, 138, 140, 140, 136, 140, 140, 135, 135, 141, 140, 140]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586238499Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"downstairs","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586270615Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"downstairs","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"[1]R flush_numbers[0..12]: [247, 242, 249, 249, 236, 249, 249, 235, 235, 250, 249, 246]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586303641Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"downstairs","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"[1]R generation[0..12]: [140, 138, 140, 140, 136, 140, 140, 135, 135, 141, 140, 140]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586336857Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"downstairs","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586368203Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"downstairs","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"[2]R flush_numbers[0..12]: [247, 242, 249, 249, 236, 249, 249, 235, 235, 250, 249, 246]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586403209Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"downstairs","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"[2]R generation[0..12]: [140, 138, 140, 140, 136, 140, 140, 135, 135, 141, 140, 140]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586440324Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"downstairs","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-11-06T20:17:26.586477089Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"downstairs","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"Max found gen is 142","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586512195Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"downstairs","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"Generation requested: 142 >= found:142","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.58654788Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"downstairs","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"Next flush: 251","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586584276Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"downstairs","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586616442Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"":"downstairs","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586650388Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586684613Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"client":"0","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586704631Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"client":"1","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586721579Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"client":"2","":"downstairs","session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586738736Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586754674Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"6a30ff3b-8357-4f43-b8a4-1c332cf2a2f2 is now active with session: d980a3f4-4596-4c20-9aa1-ab4a89c79613","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586772812Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.58678808Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771,"session_id":"d980a3f4-4596-4c20-9aa1-ab4a89c79613"} {"msg":"The guest has finished waiting for activation with:142","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586804178Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771} {"msg":"Activated sub_volume 6a30ff3b-8357-4f43-b8a4-1c332cf2a2f2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:26.586818217Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 284 len 8192 data: 9 8 02/30 Write block 293 len 20480 data: 15 14 13 11 9 03/30 Flush 04/30 Write block 254 len 8192 data: 12 11 05/30 Read block 9 len 36864 06/30 Flush 07/30 Write block 205 len 24576 data: 15 11 10 7 7 8 08/30 Write block 306 len 32768 data: 12 14 13 14 13 14 14 13 09/30 Read block 407 len 28672 10/30 Read block 15 len 28672 11/30 Write block 395 len 40960 data: 11 8 9 8 9 10 12 11 11 12 12/30 Read block 549 len 20480 13/30 Write block 577 len 40960 data: 9 8 10 7 7 7 7 6 11 11 14/30 Read block 271 len 24576 15/30 Read block 12 len 12288 16/30 Write block 207 len 24576 data: 11 8 8 9 7 7 17/30 Flush 18/30 Write block 485 len 20480 data: 11 7 8 8 7 19/30 Write block 389 len 8192 data: 8 10 20/30 Read block 117 len 4096 21/30 Write block 169 len 20480 data: 12 8 8 7 6 22/30 Write block 294 len 20480 data: 15 14 12 10 11 23/30 Read block 346 len 16384 24/30 Read block 247 len 36864 25/30 Write block 129 len 4096 data: 16 26/30 Write block 415 len 20480 data: 9 11 12 16 15 27/30 Read block 238 len 28672 28/30 Write block 132 len 16384 data: 18 18 17 14 29/30 Write block 164 len 8192 data: 12 10 30/30 Write block 27 len 8192 data: 15 14 ---------------------------------------------------------------- Crucible gen:142 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 2 Sent Done Done false 1004 Acked Read 9 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 6 Sent Done Done false 1007 Acked Write 8 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Write 10 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Write 10 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Read 1 Sent Done Done false 1020 Acked Write 5 Sent Done Done false 1021 Acked Write 5 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Read 9 Sent Done Done false 1024 Acked Write 1 Sent Done Done false 1025 Acked Write 5 Sent Done Done false 1026 Acked Read 7 Sent Done Done false 1027 Acked Write 4 Sent Sent Sent false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 3 3 33 Done 0 24 24 48 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:26.605393056Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1771} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 247 247 247 F F F 1 020-039 138 138 138 242 242 242 F T T <--- 2 040-059 140 140 140 249 249 249 F F F 3 060-079 140 140 140 249 249 249 F F F 4 080-099 136 136 136 236 236 236 F F F 5 100-119 140 140 140 249 249 249 F F F 6 120-139 140 140 140 249 249 249 F T T <--- 7 140-159 135 135 135 235 235 235 F F F 8 160-179 135 135 135 235 235 235 F T T <--- 9 180-199 141 141 141 250 250 250 F F F 10 200-219 140 142 142 249 253 253 F F F <--- 11 220-239 140 140 140 246 246 246 F F F 12 240-259 138 142 142 242 252 252 F F F <--- 13 260-279 138 138 138 243 243 243 F F F 14 280-299 142 142 142 251 251 251 F T T <--- 15 300-319 141 142 142 250 253 253 F F F <--- 16 320-339 138 138 138 241 241 241 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 129 142 142 225 253 253 F T T <--- 20 400-419 134 142 142 234 253 253 F T T <--- 21 420-439 136 136 136 238 238 238 F F F 22 440-459 138 138 138 242 242 242 F F F 23 460-479 138 138 138 242 242 242 F F F 24 480-499 139 139 139 244 244 244 F T T <--- 25 500-519 139 139 139 244 244 244 F F F 26 520-539 141 141 141 250 250 250 F F F 27 540-559 141 141 141 250 250 250 F F F 28 560-579 131 142 142 228 253 253 F F F <--- 29 580-599 123 142 142 215 253 253 F F F <--- Max gen: 142, Max flush: 253 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 71 Verifying data now No change after verify Loop: 71 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 247 247 247 F F F 1 020-039 143 143 143 254 254 254 F F F 2 040-059 140 140 140 249 249 249 F F F 3 060-079 140 140 140 249 249 249 F F F 4 080-099 136 136 136 236 236 236 F F F 5 100-119 140 140 140 249 249 249 F F F 6 120-139 143 143 143 254 254 254 F F F 7 140-159 135 135 135 235 235 235 F F F 8 160-179 143 143 143 254 254 254 F F F 9 180-199 141 141 141 250 250 250 F F F 10 200-219 142 142 142 253 253 253 F F F 11 220-239 140 140 140 246 246 246 F F F 12 240-259 142 142 142 252 252 252 F F F 13 260-279 138 138 138 243 243 243 F F F 14 280-299 143 143 143 254 254 254 F F F 15 300-319 142 142 142 253 253 253 F F F 16 320-339 138 138 138 241 241 241 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 143 143 143 254 254 254 F F F 20 400-419 143 143 143 254 254 254 F F F 21 420-439 136 136 136 238 238 238 F F F 22 440-459 138 138 138 242 242 242 F F F 23 460-479 138 138 138 242 242 242 F F F 24 480-499 143 143 143 254 254 254 F F F 25 500-519 139 139 139 244 244 244 F F F 26 520-539 141 141 141 250 250 250 F F F 27 540-559 141 141 141 250 250 250 F F F 28 560-579 142 142 142 253 253 253 F F F 29 580-599 142 142 142 253 253 253 F F F Max gen: 143, Max flush: 254 Begin loop 72 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.042126961Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:31.085776862Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.08618442Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.117684062Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.118274537Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.118327001Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.118344388Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.118366466Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779} {"msg":"Crucible a92fc2cf-6d3c-4806-b925-a136133d2100 has session id: 8ca41500-ab88-46ca-8663-f1d8241eb67a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.118876841Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"Upstairs opts: Upstairs UUID: a92fc2cf-6d3c-4806-b925-a136133d2100, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.118922795Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"Crucible stats registered with UUID: a92fc2cf-6d3c-4806-b925-a136133d2100","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.118943663Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:31.118959911Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.119159516Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"io task","client":"0","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.119191602Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"io task","client":"1","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.119210019Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"io task","client":"2","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.11936625Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"task":"crutest"} {"msg":"The guest has requested activation with gen:144","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.119390827Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779} {"msg":"a92fc2cf-6d3c-4806-b925-a136133d2100 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.119466947Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.119486235Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"client":"0","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.119506352Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"client":"1","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.11952319Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"client":"2","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.119539658Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"io task","client":"2","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.119556936Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"io task","client":"0","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.119573833Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"io task","client":"1","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"ds_connection connected from Ok(127.0.0.1:45224)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.119720835Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"io task","client":"2","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"ds_connection connected from Ok(127.0.0.1:54101)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.11976448Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"io task","client":"1","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"ds_connection connected from Ok(127.0.0.1:41990)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.119810864Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"io task","client":"0","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.120970688Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"client":"2","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121090582Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"client":"0","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121118279Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"client":"1","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121565702Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121692606Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"[0]R flush_numbers[0..12]: [247, 254, 249, 249, 236, 249, 254, 235, 254, 250, 253, 246]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121724802Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"downstairs","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"[0]R generation[0..12]: [140, 143, 140, 140, 136, 140, 143, 135, 143, 141, 142, 140]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.12174613Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"downstairs","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121771057Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"downstairs","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"[1]R flush_numbers[0..12]: [247, 254, 249, 249, 236, 249, 254, 235, 254, 250, 253, 246]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121790974Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"downstairs","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"[1]R generation[0..12]: [140, 143, 140, 140, 136, 140, 143, 135, 143, 141, 142, 140]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121809482Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"downstairs","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121829349Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"downstairs","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"[2]R flush_numbers[0..12]: [247, 254, 249, 249, 236, 249, 254, 235, 254, 250, 253, 246]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121848367Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"downstairs","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"[2]R generation[0..12]: [140, 143, 140, 140, 136, 140, 143, 135, 143, 141, 142, 140]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121867055Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"downstairs","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121886672Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"downstairs","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"Max found gen is 144","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.12190374Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"downstairs","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"Generation requested: 144 >= found:144","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121920628Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"downstairs","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"Next flush: 255","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121936456Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"downstairs","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121959333Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"":"downstairs","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121976441Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.121991629Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"client":"0","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.122008407Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"client":"1","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.122026904Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"client":"2","":"downstairs","session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.122044212Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.12206028Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"a92fc2cf-6d3c-4806-b925-a136133d2100 is now active with session: 8ca41500-ab88-46ca-8663-f1d8241eb67a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.122078318Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.122093216Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779,"session_id":"8ca41500-ab88-46ca-8663-f1d8241eb67a"} {"msg":"The guest has finished waiting for activation with:144","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.122109194Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779} {"msg":"Activated sub_volume a92fc2cf-6d3c-4806-b925-a136133d2100","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:31.122122912Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 141 len 4096 data: 11 02/30 Read block 565 len 24576 03/30 Read block 66 len 40960 04/30 Write block 368 len 32768 data: 8 7 6 7 6 5 6 5 05/30 Read block 147 len 40960 06/30 Write block 450 len 20480 data: 21 20 16 17 18 07/30 Write block 577 len 32768 data: 10 9 11 8 8 8 8 7 08/30 Read block 275 len 36864 09/30 Read block 243 len 36864 10/30 Read block 470 len 12288 11/30 Read block 101 len 32768 12/30 Write block 571 len 12288 data: 8 7 8 13/30 Read block 14 len 32768 14/30 Read block 501 len 28672 15/30 Read block 570 len 36864 16/30 Write block 102 len 24576 data: 10 8 9 10 10 12 17/30 Read block 141 len 4096 18/30 Write block 376 len 12288 data: 5 7 7 19/30 Read block 70 len 40960 20/30 Read block 161 len 28672 21/30 Write block 408 len 36864 data: 10 12 12 13 12 11 8 10 12 22/30 Read block 48 len 24576 23/30 Read block 518 len 12288 24/30 Read block 322 len 40960 25/30 Read block 125 len 16384 26/30 Read block 50 len 20480 27/30 Write block 190 len 20480 data: 9 11 11 10 10 28/30 Write block 271 len 20480 data: 13 13 13 12 13 29/30 Read block 22 len 8192 30/30 Write block 290 len 8192 data: 16 14 ---------------------------------------------------------------- Crucible gen:144 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 6 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Write 8 Done Done Done false 1004 Acked Read 10 Done Done Done false 1005 Acked Write 5 Done Done Done false 1006 Acked Write 8 Done Done Done false 1007 Acked Read 9 Done Done Done false 1008 Acked Read 9 Done Done Done false 1009 Acked Read 3 Done Done Done false 1010 Acked Read 8 Done Done Done false 1011 Acked Write 3 Done Done Done false 1012 Acked Read 8 Done Done Done false 1013 Acked Read 7 Done Done Done false 1014 Acked Read 9 Done Done Done false 1015 Acked Write 6 Done Done Done false 1016 Acked Read 1 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Read 10 Sent Done Done false 1019 Acked Read 7 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Read 6 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Read 10 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Read 5 Sent Done Done false 1026 Acked Write 5 Sent Done Done false 1027 Acked Write 5 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 14 1 1 16 Done 16 29 29 74 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:31.143031378Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1779} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 247 247 247 F F F 1 020-039 143 143 143 254 254 254 F F F 2 040-059 140 140 140 249 249 249 F F F 3 060-079 140 140 140 249 249 249 F F F 4 080-099 136 136 136 236 236 236 F F F 5 100-119 140 140 140 249 249 249 T T T 6 120-139 143 143 143 254 254 254 F F F 7 140-159 135 135 135 235 235 235 T T T 8 160-179 143 143 143 254 254 254 F F F 9 180-199 141 141 141 250 250 250 F T T <--- 10 200-219 142 142 142 253 253 253 F F F 11 220-239 140 140 140 246 246 246 F F F 12 240-259 142 142 142 252 252 252 F F F 13 260-279 138 138 138 243 243 243 F T T <--- 14 280-299 143 143 143 254 254 254 F T T <--- 15 300-319 142 142 142 253 253 253 F F F 16 320-339 138 138 138 241 241 241 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 132 132 132 230 230 230 T T T 19 380-399 143 143 143 254 254 254 F F F 20 400-419 143 143 143 254 254 254 F T T <--- 21 420-439 136 136 136 238 238 238 F F F 22 440-459 138 138 138 242 242 242 T T T 23 460-479 138 138 138 242 242 242 F F F 24 480-499 143 143 143 254 254 254 F F F 25 500-519 139 139 139 244 244 244 F F F 26 520-539 141 141 141 250 250 250 F F F 27 540-559 141 141 141 250 250 250 F F F 28 560-579 142 142 142 253 253 253 T T T 29 580-599 142 142 142 253 253 253 T T T Max gen: 143, Max flush: 254 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 72 Verifying data now Files /tmp/test_live_repair/test_repair_verify.data and /tmp/test_live_repair/test_repair_verify.data.last differ 379,381c379 < 4, < 6, < 6, --- > 5, 382a381,382 > 7, > 7, diff found after verify Loop: 72 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 140 140 247 247 247 F F F 1 020-039 143 143 143 254 254 254 F F F 2 040-059 140 140 140 249 249 249 F F F 3 060-079 140 140 140 249 249 249 F F F 4 080-099 136 136 136 236 236 236 F F F 5 100-119 144 144 144 255 255 255 F F F 6 120-139 143 143 143 254 254 254 F F F 7 140-159 144 144 144 255 255 255 F F F 8 160-179 143 143 143 254 254 254 F F F 9 180-199 144 144 144 255 255 255 F F F 10 200-219 142 142 142 253 253 253 F F F 11 220-239 140 140 140 246 246 246 F F F 12 240-259 142 142 142 252 252 252 F F F 13 260-279 144 144 144 255 255 255 F F F 14 280-299 144 144 144 255 255 255 F F F 15 300-319 142 142 142 253 253 253 F F F 16 320-339 138 138 138 241 241 241 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 144 144 144 255 255 255 F F F 19 380-399 143 143 143 254 254 254 F F F 20 400-419 144 144 144 255 255 255 F F F 21 420-439 136 136 136 238 238 238 F F F 22 440-459 144 144 144 255 255 255 F F F 23 460-479 138 138 138 242 242 242 F F F 24 480-499 143 143 143 254 254 254 F F F 25 500-519 139 139 139 244 244 244 F F F 26 520-539 141 141 141 250 250 250 F F F 27 540-559 141 141 141 250 250 250 F F F 28 560-579 144 144 144 255 255 255 F F F 29 580-599 144 144 144 255 255 255 F F F Max gen: 144, Max flush: 255 Begin loop 73 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.579843653Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:35.623928029Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.624328779Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.655771067Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.656323698Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.656378181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.656397898Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.656425675Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790} {"msg":"Crucible 713782e8-6b54-4aa5-a10c-0d0a8619b487 has session id: cebe4a0f-f970-4492-bb50-5ae23c18dfd8","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.656970146Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"Upstairs opts: Upstairs UUID: 713782e8-6b54-4aa5-a10c-0d0a8619b487, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657064764Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"Crucible stats registered with UUID: 713782e8-6b54-4aa5-a10c-0d0a8619b487","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.65709626Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:35.657122947Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657260919Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"io task","client":"2","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657306994Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657331361Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657477932Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"task":"crutest"} {"msg":"The guest has requested activation with gen:146","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657501079Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790} {"msg":"713782e8-6b54-4aa5-a10c-0d0a8619b487 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.65757347Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657593808Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"client":"0","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657611175Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"client":"1","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657628853Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"client":"2","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657646181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"io task","client":"2","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657674107Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657706443Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"ds_connection connected from Ok(127.0.0.1:42940)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657745109Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"io task","client":"2","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"ds_connection connected from Ok(127.0.0.1:47949)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.657779284Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"ds_connection connected from Ok(127.0.0.1:46338)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.65781292Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.659033986Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"client":"2","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.65908166Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"client":"1","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.65924353Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"client":"0","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.65955982Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.659616403Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"[0]R flush_numbers[0..12]: [247, 254, 249, 249, 236, 255, 254, 255, 254, 255, 253, 246]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.659810348Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"downstairs","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"[0]R generation[0..12]: [140, 143, 140, 140, 136, 144, 143, 144, 143, 144, 142, 140]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.659835145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"downstairs","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.659855503Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"downstairs","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"[1]R flush_numbers[0..12]: [247, 254, 249, 249, 236, 255, 254, 255, 254, 255, 253, 246]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.6598749Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"downstairs","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"[1]R generation[0..12]: [140, 143, 140, 140, 136, 144, 143, 144, 143, 144, 142, 140]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.659894168Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"downstairs","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.659914435Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"downstairs","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"[2]R flush_numbers[0..12]: [247, 254, 249, 249, 236, 255, 254, 255, 254, 255, 253, 246]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.659936933Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"downstairs","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"[2]R generation[0..12]: [140, 143, 140, 140, 136, 144, 143, 144, 143, 144, 142, 140]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.65995645Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"downstairs","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.659976417Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"downstairs","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"Max found gen is 145","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.659992915Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"downstairs","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 146 >= found:145","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.660017782Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"downstairs","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"Next flush: 256","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.66003415Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"downstairs","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.660050268Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"":"downstairs","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.660067486Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.660082944Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"client":"0","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.660100172Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"client":"1","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.660132898Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"client":"2","":"downstairs","session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.660151166Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.660167403Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"713782e8-6b54-4aa5-a10c-0d0a8619b487 is now active with session: cebe4a0f-f970-4492-bb50-5ae23c18dfd8","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.660185351Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"session_id":"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.660200649Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790,"session_id"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :"cebe4a0f-f970-4492-bb50-5ae23c18dfd8"} {"msg":"The guest has finished waiting for activation with:146","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.660223976Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790} {"msg":"Activated sub_volume 713782e8-6b54-4aa5-a10c-0d0a8619b487","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:35.660237915Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 233 len 36864 data: 12 9 11 11 11 13 14 13 14 02/30 Read block 557 len 28672 03/30 Write block 5 len 24576 data: 4 6 8 9 9 10 04/30 Read block 59 len 32768 05/30 Read block 348 len 8192 06/30 Read block 313 len 20480 07/30 Read block 52 len 16384 08/30 Read block 138 len 24576 09/30 Write block 318 len 16384 data: 9 10 9 11 10/30 Read block 313 len 8192 11/30 Write block 118 len 36864 data: 16 17 18 13 12 14 14 14 13 12/30 Flush 13/30 Write block 165 len 16384 data: 11 11 11 12 14/30 Read block 522 len 32768 15/30 Read block 514 len 8192 16/30 Write block 568 len 16384 data: 8 7 8 9 17/30 Write block 511 len 24576 data: 10 11 10 10 11 11 18/30 Read block 383 len 12288 19/30 Write block 385 len 20480 data: 12 10 7 8 9 20/30 Read block 162 len 28672 21/30 Write block 149 len 36864 data: 6 5 5 6 7 9 10 8 10 22/30 Write block 336 len 12288 data: 14 15 14 23/30 Read block 390 len 16384 24/30 Read block 575 len 12288 25/30 Read block 398 len 12288 26/30 Write block 61 len 4096 data: 7 27/30 Read block 46 len 12288 28/30 Write block 561 len 8192 data: 8 8 29/30 Read block 337 len 12288 30/30 Read block 283 len 16384 ---------------------------------------------------------------- Crucible gen:146 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Sent Done Done false 1001 Acked Read 7 Sent Done Done false 1002 Acked Write 6 Sent Done Done false 1003 Acked Read 8 Sent Done Done false 1004 Acked Read 2 Sent Done Done false 1005 Acked Read 5 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Read 6 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Write 9 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 4 Sent Done Done false 1013 Acked Read 8 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Write 4 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Write 5 Sent Done Done false 1019 Acked Read 7 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Read 3 Sent Done Done false 1027 Acked Write 2 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Read 4 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 0 31 Done 0 29 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1011)) Some(JobId(1011)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:35.679387192Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1790} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 140 146 146 247 256 256 F F F <--- 1 020-039 143 143 143 254 254 254 F F F 2 040-059 140 140 140 249 249 249 F F F 3 060-079 140 140 140 249 249 249 F T T <--- 4 080-099 136 136 136 236 236 236 F F F 5 100-119 144 146 146 255 256 256 F F F <--- 6 120-139 143 146 146 254 256 256 F F F <--- 7 140-159 144 144 144 255 255 255 F T T <--- 8 160-179 143 143 143 254 254 254 F T T <--- 9 180-199 144 144 144 255 255 255 F F F 10 200-219 142 142 142 253 253 253 F F F 11 220-239 140 146 146 246 256 256 F F F <--- 12 240-259 142 146 146 252 256 256 F F F <--- 13 260-279 144 144 144 255 255 255 F F F 14 280-299 144 144 144 255 255 255 F F F 15 300-319 142 146 146 253 256 256 F F F <--- 16 320-339 138 146 146 241 256 256 F T T <--- 17 340-359 132 132 132 232 232 232 F F F 18 360-379 144 144 144 255 255 255 F F F 19 380-399 143 143 143 254 254 254 F T T <--- 20 400-419 144 144 144 255 255 255 F F F 21 420-439 136 136 136 238 238 238 F F F 22 440-459 144 144 144 255 255 255 F F F 23 460-479 138 138 138 242 242 242 F F F 24 480-499 143 143 143 254 254 254 F F F 25 500-519 139 139 139 244 244 244 F T T <--- 26 520-539 141 141 141 250 250 250 F F F 27 540-559 141 141 141 250 250 250 F F F 28 560-579 144 144 144 255 255 255 F T T <--- 29 580-599 144 144 144 255 255 255 F F F Max gen: 146, Max flush: 256 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 73 Verifying data now No change after verify Loop: 73 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 256 256 256 F F F 1 020-039 143 143 143 254 254 254 F F F 2 040-059 140 140 140 249 249 249 F F F 3 060-079 147 147 147 257 257 257 F F F 4 080-099 136 136 136 236 236 236 F F F 5 100-119 146 146 146 256 256 256 F F F 6 120-139 146 146 146 256 256 256 F F F 7 140-159 147 147 147 257 257 257 F F F 8 160-179 147 147 147 257 257 257 F F F 9 180-199 144 144 144 255 255 255 F F F 10 200-219 142 142 142 253 253 253 F F F 11 220-239 146 146 146 256 256 256 F F F 12 240-259 146 146 146 256 256 256 F F F 13 260-279 144 144 144 255 255 255 F F F 14 280-299 144 144 144 255 255 255 F F F 15 300-319 146 146 146 256 256 256 F F F 16 320-339 147 147 147 257 257 257 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 144 144 144 255 255 255 F F F 19 380-399 147 147 147 257 257 257 F F F 20 400-419 144 144 144 255 255 255 F F F 21 420-439 136 136 136 238 238 238 F F F 22 440-459 144 144 144 255 255 255 F F F 23 460-479 138 138 138 242 242 242 F F F 24 480-499 143 143 143 254 254 254 F F F 25 500-519 147 147 147 257 257 257 F F F 26 520-539 141 141 141 250 250 250 F F F 27 540-559 141 141 141 250 250 250 F F F 28 560-579 147 147 147 257 257 257 F F F 29 580-599 144 144 144 255 255 255 F F F Max gen: 147, Max flush: 257 Begin loop 74 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.139187781Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.184764839Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.185405578Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.185455152Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.185471579Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.185493847Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798} {"msg":"Crucible 8945209d-af02-4a27-9b4a-16cbe09d4590 has session id: 0327becf-9fc4-438e-80cf-7ef024d7b032","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.186040178Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"Upstairs opts: Upstairs UUID: 8945209d-af02-4a27-9b4a-16cbe09d4590, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.186097691Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"Crucible stats registered with UUID: 8945209d-af02-4a27-9b4a-16cbe09d4590","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.186121448Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:40.186138525Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.186393673Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"io task","client":"0","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.186440457Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.186467594Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"io task","client":"1","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.186792693Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"task":"crutest"} {"msg":"The guest has requested activation with gen:148","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.186829958Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798} {"msg":"8945209d-af02-4a27-9b4a-16cbe09d4590 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.1868948Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.186919637Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"client":"0","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.186939325Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"client":"1","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.186968421Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"client":"2","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.187000717Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.187033173Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"io task","client":"0","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.187062859Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"io task","client":"1","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"ds_connection connected from Ok(127.0.0.1:33264)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.187111583Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"ds_connection connected from Ok(127.0.0.1:45171)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.187162376Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"io task","client":"1","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"ds_connection connected from Ok(127.0.0.1:33202)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.187407486Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"io task","client":"0","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.188347447Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"client":"0","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.188395891Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"client":"1","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.188511586Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"client":"2","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.188909806Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189003354Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"[0]R flush_numbers[0..12]: [256, 254, 249, 257, 236, 256, 256, 257, 257, 255, 253, 256]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189112601Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"downstairs","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"[0]R generation[0..12]: [146, 143, 140, 147, 136, 146, 146, 147, 147, 144, 142, 146]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189138777Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"downstairs","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189159425Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"downstairs","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"[1]R flush_numbers[0..12]: [256, 254, 249, 257, 236, 256, 256, 257, 257, 255, 253, 256]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189179092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"downstairs","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"[1]R generation[0..12]: [146, 143, 140, 147, 136, 146, 146, 147, 147, 144, 142, 146]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.18919816Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"downstairs","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189218047Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"downstairs","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"[2]R flush_numbers[0..12]: [256, 254, 249, 257, 236, 256, 256, 257, 257, 255, 253, 256]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189237025Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"downstairs","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"[2]R generation[0..12]: [146, 143, 140, 147, 136, 146, 146, 147, 147, 144, 142, 146]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189255823Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"downstairs","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.18927542Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"downstairs","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"Max found gen is 148","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189291628Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"Generation requested: 148 >= found:148","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189315215Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"downstairs","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"Next flush: 258","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189331323Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"downstairs","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189352321Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"":"downstairs","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189370088Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189385196Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"client":"0","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189402274Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"client":"1","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189419112Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"client":"2","":"downstairs","session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.18943631Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189452548Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"8945209d-af02-4a27-9b4a-16cbe09d4590 is now active with session: 0327becf-9fc4-438e-80cf-7ef024d7b032","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189471595Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189486953Z","hostname":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798,"session_id":"0327becf-9fc4-438e-80cf-7ef024d7b032"} {"msg":"The guest has finished waiting for activation with:148","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189507261Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798} {"msg":"Activated sub_volume 8945209d-af02-4a27-9b4a-16cbe09d4590","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:40.189521219Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 589 len 36864 data: 18 18 15 15 12 11 11 11 6 02/30 Read block 102 len 36864 03/30 Write block 245 len 12288 data: 18 16 13 04/30 Flush 05/30 Write block 398 len 24576 data: 9 10 11 13 12 12 06/30 Write block 544 len 24576 data: 15 13 10 11 12 11 07/30 Flush 08/30 Write block 274 len 24576 data: 13 14 11 13 12 11 09/30 Read block 24 len 32768 10/30 Read block 160 len 12288 11/30 Write block 97 len 20480 data: 12 11 12 9 9 12/30 Write block 232 len 28672 data: 12 13 10 12 12 12 14 13/30 Write block 91 len 24576 data: 13 11 10 10 11 11 14/30 Flush 15/30 Write block 585 len 16384 data: 12 12 12 15 16/30 Write block 518 len 12288 data: 11 10 10 17/30 Read block 213 len 4096 18/30 Read block 153 len 32768 19/30 Write block 376 len 20480 data: 5 7 7 8 9 20/30 Write block 361 len 8192 data: 9 7 21/30 Read block 567 len 40960 22/30 Write block 176 len 24576 data: 10 12 13 18 20 21 23/30 Write block 569 len 28672 data: 8 9 10 8 9 7 8 24/30 Read block 520 len 16384 25/30 Write block 200 len 36864 data: 13 15 13 13 13 16 12 12 9 26/30 Flush 27/30 Write block 108 len 12288 data: 13 13 16 28/30 Read block 10 len 16384 29/30 Read block 142 len 40960 30/30 Read block 106 len 20480 ---------------------------------------------------------------- Crucible gen:148 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Sent false 1001 Acked Read 9 Done Done Sent false 1002 Acked Write 3 Done Done Sent false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Write 6 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 6 Done Done Sent false 1008 Acked Read 8 Done Done Sent false 1009 Acked Read 3 Done Done Sent false 1010 Acked Write 5 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Write 3 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Write 5 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Read 10 Done Done Sent false 1021 Acked Write 6 Done Done Sent false 1022 Acked Write 7 Done Done Sent false 1023 Acked Read 4 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 3 Done Done Sent false 1027 Acked Read 4 Done Done Sent false 1028 Acked Read 10 Done Done Sent false 1029 Acked Read 5 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:40.209730033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1798} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 256 256 256 F F F 1 020-039 143 143 143 254 254 254 F F F 2 040-059 140 140 140 249 249 249 F F F 3 060-079 147 147 147 257 257 257 F F F 4 080-099 148 148 136 260 260 236 F F F <--- 5 100-119 148 148 146 260 260 256 T T F <--- 6 120-139 146 146 146 256 256 256 F F F 7 140-159 147 147 147 257 257 257 F F F 8 160-179 148 148 147 261 261 257 F F F <--- 9 180-199 148 148 144 261 261 255 F F F <--- 10 200-219 148 148 142 261 261 253 F F F <--- 11 220-239 148 148 146 260 260 256 F F F <--- 12 240-259 148 148 146 258 258 256 F F F <--- 13 260-279 148 148 144 260 260 255 F F F <--- 14 280-299 144 144 144 255 255 255 F F F 15 300-319 146 146 146 256 256 256 F F F 16 320-339 147 147 147 257 257 257 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 148 148 144 261 261 255 F F F <--- 19 380-399 148 148 147 261 261 257 F F F <--- 20 400-419 148 148 144 259 259 255 F F F <--- 21 420-439 136 136 136 238 238 238 F F F 22 440-459 144 144 144 255 255 255 F F F 23 460-479 138 138 138 242 242 242 F F F 24 480-499 143 143 143 254 254 254 F F F 25 500-519 148 148 147 261 261 257 F F F <--- 26 520-539 148 148 141 261 261 250 F F F <--- 27 540-559 148 148 141 259 259 250 F F F <--- 28 560-579 148 148 147 261 261 257 F F F <--- 29 580-599 148 148 144 261 261 255 F F F <--- Max gen: 148, Max flush: 261 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 74 Verifying data now No change after verify Loop: 74 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 256 256 256 F F F 1 020-039 143 143 143 254 254 254 F F F 2 040-059 140 140 140 249 249 249 F F F 3 060-079 147 147 147 257 257 257 F F F 4 080-099 148 148 148 260 260 260 F F F 5 100-119 149 149 149 262 262 262 F F F 6 120-139 146 146 146 256 256 256 F F F 7 140-159 147 147 147 257 257 257 F F F 8 160-179 148 148 148 261 261 261 F F F 9 180-199 148 148 148 261 261 261 F F F 10 200-219 148 148 148 261 261 261 F F F 11 220-239 148 148 148 260 260 260 F F F 12 240-259 148 148 148 258 258 258 F F F 13 260-279 148 148 148 260 260 260 F F F 14 280-299 144 144 144 255 255 255 F F F 15 300-319 146 146 146 256 256 256 F F F 16 320-339 147 147 147 257 257 257 F F F 17 340-359 132 132 132 232 232 232 F F F 18 360-379 148 148 148 261 261 261 F F F 19 380-399 148 148 148 261 261 261 F F F 20 400-419 148 148 148 259 259 259 F F F 21 420-439 136 136 136 238 238 238 F F F 22 440-459 144 144 144 255 255 255 F F F 23 460-479 138 138 138 242 242 242 F F F 24 480-499 143 143 143 254 254 254 F F F 25 500-519 148 148 148 261 261 261 F F F 26 520-539 148 148 148 261 261 261 F F F 27 540-559 148 148 148 259 259 259 F F F 28 560-579 148 148 148 261 261 261 F F F 29 580-599 148 148 148 261 261 261 F F F Max gen: 149, Max flush: 262 Begin loop 75 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.623709285Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.669214212Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.66985853Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.669919363Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.66993892Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.669970736Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806} {"msg":"Crucible 4b7fa26a-160e-4e81-9654-064ef0a7ef73 has session id: 7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.670621694Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"Upstairs opts: Upstairs UUID: 4b7fa26a-160e-4e81-9654-064ef0a7ef73, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.670661699Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"Crucible stats registered with UUID: 4b7fa26a-160e-4e81-9654-064ef0a7ef73","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.670680117Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:44.670696245Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.670930805Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.67097053Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"2","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.670990808Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.67120807Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"task":"crutest"} {"msg":"The guest has requested activation with gen:150","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.671231258Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806} {"msg":"4b7fa26a-160e-4e81-9654-064ef0a7ef73 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.671300349Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.671321126Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"client":"0","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.671338434Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"client":"1","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.671354812Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"client":"2","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.67137114Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"2","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.671392207Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.671455569Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"ds_connection connected from Ok(127.0.0.1:60314)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.671505703Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"ds_connection connected from Ok(127.0.0.1:52020)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.671535989Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"2","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:44.671570575Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:44.67160522Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:44.671678861Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.671707797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.671740753Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.67272088Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"client":"0","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.672769154Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"client":"2","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.673385776Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:44.673421351Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.673142676Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"ds_connection connected from Ok(127.0.0.1:53491)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.673247932Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.674967555Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"client":"1","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"[0]R flush_numbers[0..12]: [256, 254, 249, 257, 260, 262, 256, 257, 261, 261, 261, 260]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675610905Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"downstairs","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"[0]R generation[0..12]: [146, 143, 140, 147, 148, 149, 146, 147, 148, 148, 148, 148]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675640261Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"downstairs","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675661118Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"downstairs","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"[1]R flush_numbers[0..12]: [256, 254, 249, 257, 260, 262, 256, 257, 261, 261, 261, 260]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675680346Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"downstairs","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"[1]R generation[0..12]: [146, 143, 140, 147, 148, 149, 146, 147, 148, 148, 148, 148]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675699473Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"downstairs","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675719231Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"downstairs","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"[2]R flush_numbers[0..12]: [256, 254, 249, 257, 260, 262, 256, 257, 261, 261, 261, 260]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675737949Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"downstairs","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"[2]R generation[0..12]: [146, 143, 140, 147, 148, 149, 146, 147, 148, 148, 148, 148]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675756956Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"downstairs","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675776394Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"downstairs","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"Max found gen is 150","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675792642Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"downstairs","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"Generation requested: 150 >= found:150","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675809549Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"downstairs","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"Next flush: 263","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675825778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"downstairs","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675841935Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"":"downstairs","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675862173Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675878011Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"client":"0","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675894669Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"client":"1","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675919136Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"client":"2","":"downstairs","session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675936384Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675952342Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"4b7fa26a-160e-4e81-9654-064ef0a7ef73 is now active with session: 7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675970139Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.675985187Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806,"session_id":"7d5aa74f-8dcb-4a97-8ee4-2fcab5d09ba9"} {"msg":"The guest has finished waiting for activation with:150","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.676008514Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806} {"msg":"Activated sub_volume 4b7fa26a-160e-4e81-9654-064ef0a7ef73","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:54.676028382Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 392 len 32768 data: 13 12 12 12 9 10 10 11 02/30 Read block 466 len 32768 03/30 Write block 226 len 4096 data: 6 04/30 Write block 416 len 12288 data: 13 13 17 05/30 Read block 435 len 20480 06/30 Flush 07/30 Write block 286 len 20480 data: 8 9 7 15 17 08/30 Read block 523 len 20480 09/30 Read block 194 len 16384 10/30 Read block 576 len 8192 11/30 Write block 323 len 4096 data: 9 12/30 Read block 256 len 36864 13/30 Flush 14/30 Write block 483 len 20480 data: 13 13 12 8 9 15/30 Read block 574 len 32768 16/30 Read block 233 len 24576 17/30 Write block 305 len 20480 data: 10 13 15 14 15 18/30 Read block 239 len 32768 19/30 Write block 534 len 12288 data: 12 13 14 20/30 Read block 475 len 16384 21/30 Read block 260 len 20480 22/30 Flush 23/30 Write block 114 len 4096 data: 18 24/30 Write block 467 len 28672 data: 12 10 8 12 13 15 15 25/30 Write block 153 len 40960 data: 8 10 11 9 11 10 12 12 12 13 26/30 Write block 344 len 12288 data: 8 7 8 27/30 Write block 588 len 8192 data: 16 19 28/30 Write block 208 len 40960 data: 10 9 10 8 8 8 8 6 7 7 29/30 Read block 307 len 4096 30/30 Write block 538 len 12288 data: 14 15 17 ---------------------------------------------------------------- Crucible gen:150 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 8 Done Sent Done false 1002 Acked Write 1 Done Sent Done false 1003 Acked Write 3 Done Sent Done false 1004 Acked Read 5 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 4 Done Sent Done false 1009 Acked Read 2 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 5 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Read 8 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Read 5 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 1 Done Sent Done false 1023 Acked Write 7 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 1 31 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:54.69445939Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1806} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 256 256 256 F F F 1 020-039 143 143 143 254 254 254 F F F 2 040-059 140 140 140 249 249 249 F F F 3 060-079 147 147 147 257 257 257 F F F 4 080-099 148 148 148 260 260 260 F F F 5 100-119 149 149 149 262 262 262 T F T <--- 6 120-139 146 146 146 256 256 256 F F F 7 140-159 147 147 147 257 257 257 T F T <--- 8 160-179 148 148 148 261 261 261 T F T <--- 9 180-199 148 148 148 261 261 261 F F F 10 200-219 148 148 148 261 261 261 T F T <--- 11 220-239 150 148 150 263 260 263 F F F <--- 12 240-259 148 148 148 258 258 258 F F F 13 260-279 148 148 148 260 260 260 F F F 14 280-299 150 144 150 264 255 264 F F F <--- 15 300-319 150 146 150 265 256 265 F F F <--- 16 320-339 150 147 150 264 257 264 F F F <--- 17 340-359 132 132 132 232 232 232 T F T <--- 18 360-379 148 148 148 261 261 261 F F F 19 380-399 150 148 150 263 261 263 F T F <--- 20 400-419 150 148 150 263 259 263 F F F <--- 21 420-439 136 136 136 238 238 238 F F F 22 440-459 144 144 144 255 255 255 F F F 23 460-479 138 138 138 242 242 242 T F T <--- 24 480-499 150 143 150 265 254 265 F F F <--- 25 500-519 148 148 148 261 261 261 F F F 26 520-539 150 148 150 265 261 265 T F T <--- 27 540-559 148 148 148 259 259 259 T F T <--- 28 560-579 148 148 148 261 261 261 F F F 29 580-599 148 148 148 261 261 261 T F T <--- Max gen: 150, Max flush: 265 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 75 Verifying data now No change after verify Loop: 75 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 256 256 256 F F F 1 020-039 143 143 143 254 254 254 F F F 2 040-059 140 140 140 249 249 249 F F F 3 060-079 147 147 147 257 257 257 F F F 4 080-099 148 148 148 260 260 260 F F F 5 100-119 151 151 151 266 266 266 F F F 6 120-139 146 146 146 256 256 256 F F F 7 140-159 151 151 151 266 266 266 F F F 8 160-179 151 151 151 266 266 266 F F F 9 180-199 148 148 148 261 261 261 F F F 10 200-219 151 151 151 266 266 266 F F F 11 220-239 150 150 150 263 263 263 F F F 12 240-259 148 148 148 258 258 258 F F F 13 260-279 148 148 148 260 260 260 F F F 14 280-299 150 150 150 264 264 264 F F F 15 300-319 150 150 150 265 265 265 F F F 16 320-339 150 150 150 264 264 264 F F F 17 340-359 151 151 151 266 266 266 F F F 18 360-379 148 148 148 261 261 261 F F F 19 380-399 150 150 150 263 263 263 F F F 20 400-419 150 150 150 263 263 263 F F F 21 420-439 136 136 136 238 238 238 F F F 22 440-459 144 144 144 255 255 255 F F F 23 460-479 151 151 151 266 266 266 F F F 24 480-499 150 150 150 265 265 265 F F F 25 500-519 148 148 148 261 261 261 F F F 26 520-539 151 151 151 266 266 266 F F F 27 540-559 151 151 151 266 266 266 F F F 28 560-579 148 148 148 261 261 261 F F F 29 580-599 151 151 151 266 266 266 F F F Max gen: 151, Max flush: 266 Begin loop 76 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.118261552Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:59.162915046Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.163304777Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.195209208Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.195761208Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.195817401Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.195842398Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.195870914Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814} {"msg":"Crucible 7c482573-d221-4e54-8c47-9fcbadb6b133 has session id: b3a21c03-1134-44a8-a565-76bc9d36f125","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.1964602Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"Upstairs opts: Upstairs UUID: 7c482573-d221-4e54-8c47-9fcbadb6b133, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.196515413Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"Crucible stats registered with UUID: 7c482573-d221-4e54-8c47-9fcbadb6b133","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.19653934Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:59.196559847Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.196747734Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"io task","client":"1","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.19678255Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.196801277Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"io task","client":"2","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.196960407Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"task":"crutest"} {"msg":"The guest has requested activation with gen:152","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.196981274Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814} {"msg":"7c482573-d221-4e54-8c47-9fcbadb6b133 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.197054735Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.197074373Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"client":"0","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.19709162Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"client":"1","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.197108778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"client":"2","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.197130566Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"io task","client":"2","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.197148284Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.1971744Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"io task","client":"1","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"ds_connection connected from Ok(127.0.0.1:62277)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.197245741Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"io task","client":"2","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"ds_connection connected from Ok(127.0.0.1:35093)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.197292775Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"io task","client":"1","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"ds_connection connected from Ok(127.0.0.1:57930)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.197326241Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.19844171Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"client":"2","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.198475666Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"client":"0","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.198502513Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"client":"1","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199036026Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199129124Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"[0]R flush_numbers[0..12]: [256, 254, 249, 257, 260, 266, 256, 266, 266, 261, 266, 263]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199203594Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"downstairs","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"[0]R generation[0..12]: [146, 143, 140, 147, 148, 151, 146, 151, 151, 148, 151, 150]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199227471Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"downstairs","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199247519Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"downstairs","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"[1]R flush_numbers[0..12]: [256, 254, 249, 257, 260, 266, 256, 266, 266, 261, 266, 263]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199271266Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"downstairs","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"[1]R generation[0..12]: [146, 143, 140, 147, 148, 151, 146, 151, 151, 148, 151, 150]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199290903Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"downstairs","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199310601Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"downstairs","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"[2]R flush_numbers[0..12]: [256, 254, 249, 257, 260, 266, 256, 266, 266, 261, 266, 263]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199329849Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"downstairs","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"[2]R generation[0..12]: [146, 143, 140, 147, 148, 151, 146, 151, 151, 148, 151, 150]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199348896Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"downstairs","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199368604Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"downstairs","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"Max found gen is 152","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199384912Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"downstairs","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 152 >= found:152","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199408289Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"downstairs","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"Next flush: 267","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199424547Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"downstairs","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199440935Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"":"downstairs","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199457853Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.19947294Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"client":"0","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199490068Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"client":"1","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199507126Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"client":"2","":"downstairs","session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:17:59.199524234Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199545191Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"7c482573-d221-4e54-8c47-9fcbadb6b133 is now active with session: b3a21c03-1134-44a8-a565-76bc9d36f125","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199563349Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199578717Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814,"session_id":"b3a21c03-1134-44a8-a565-76bc9d36f125"} {"msg":"The guest has finished waiting for activation with:152","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199594995Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814} {"msg":"Activated sub_volume 7c482573-d221-4e54-8c47-9fcbadb6b133","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:17:59.199608953Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 73 len 28672 data: 4 7 8 10 11 15 13 03/30 Write block 84 len 36864 data: 12 8 9 9 10 13 12 14 12 04/30 Write block 198 len 28672 data: 14 13 14 16 14 14 14 05/30 Read block 119 len 16384 06/30 Read block 546 len 36864 07/30 Read block 446 len 24576 08/30 Write block 461 len 16384 data: 9 10 10 13 09/30 Read block 352 len 20480 10/30 Read block 569 len 16384 11/30 Read block 428 len 24576 12/30 Read block 386 len 24576 13/30 Flush 14/30 Write block 428 len 12288 data: 12 11 12 15/30 Read block 416 len 24576 16/30 Read block 110 len 8192 17/30 Write block 560 len 24576 data: 8 9 9 8 11 9 18/30 Write block 413 len 32768 data: 12 9 11 14 14 18 16 15 19/30 Read block 239 len 32768 20/30 Write block 286 len 36864 data: 9 10 8 16 18 15 15 16 16 21/30 Read block 451 len 16384 22/30 Read block 33 len 4096 23/30 Read block 270 len 8192 24/30 Write block 20 len 12288 data: 10 10 15 25/30 Read block 149 len 16384 26/30 Write block 92 len 32768 data: 13 11 11 12 12 13 12 13 27/30 Read block 140 len 8192 28/30 Read block 211 len 36864 29/30 Read block 492 len 28672 30/30 Read block 90 len 36864 ---------------------------------------------------------------- Crucible gen:152 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 7 Done Done Done false 1002 Acked Write 9 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Read 4 Sent Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Read 6 Sent Done Done false 1007 Acked Write 4 Sent Done Done false 1008 Acked Read 5 Sent Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Read 6 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 3 Sent Done Done false 1014 Acked Read 6 Sent Done Done false 1015 Acked Read 2 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Read 8 Sent Done Done false 1019 Acked Write 9 Sent Done Done false 1020 Acked Read 4 Sent Done Done false 1021 Acked Read 1 Sent Done Done false 1022 Acked Read 2 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 4 Sent Done Done false 1025 Acked Write 8 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Read 9 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 0 29 Done 1 28 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1012)) Some(JobId(1012)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:17:59.219645849Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1814} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 256 256 256 F F F 1 020-039 143 143 143 254 254 254 F T T <--- 2 040-059 140 140 140 249 249 249 F F F 3 060-079 147 152 152 257 268 268 T F F <--- 4 080-099 148 152 152 260 268 268 F T T <--- 5 100-119 151 151 151 266 266 266 F F F 6 120-139 146 146 146 256 256 256 F F F 7 140-159 151 151 151 266 266 266 F F F 8 160-179 151 151 151 266 266 266 F F F 9 180-199 148 152 152 261 268 268 F F F <--- 10 200-219 151 152 152 266 268 268 F F F <--- 11 220-239 150 150 150 263 263 263 F F F 12 240-259 148 148 148 258 258 258 F F F 13 260-279 148 148 148 260 260 260 F F F 14 280-299 150 150 150 264 264 264 F T T <--- 15 300-319 150 150 150 265 265 265 F F F 16 320-339 150 150 150 264 264 264 F F F 17 340-359 151 151 151 266 266 266 F F F 18 360-379 148 148 148 261 261 261 F F F 19 380-399 150 150 150 263 263 263 F F F 20 400-419 150 150 150 263 263 263 F T T <--- 21 420-439 136 136 136 238 238 238 F T T <--- 22 440-459 144 144 144 255 255 255 F F F 23 460-479 151 152 152 266 268 268 F F F <--- 24 480-499 150 150 150 265 265 265 F F F 25 500-519 148 148 148 261 261 261 F F F 26 520-539 151 151 151 266 266 266 F F F 27 540-559 151 151 151 266 266 266 F F F 28 560-579 148 148 148 261 261 261 F T T <--- 29 580-599 151 151 151 266 266 266 F F F Max gen: 152, Max flush: 268 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 76 Verifying data now No change after verify Loop: 76 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 256 256 256 F F F 1 020-039 153 153 153 269 269 269 F F F 2 040-059 140 140 140 249 249 249 F F F 3 060-079 152 152 152 268 268 268 F F F 4 080-099 153 153 153 269 269 269 F F F 5 100-119 151 151 151 266 266 266 F F F 6 120-139 146 146 146 256 256 256 F F F 7 140-159 151 151 151 266 266 266 F F F 8 160-179 151 151 151 266 266 266 F F F 9 180-199 152 152 152 268 268 268 F F F 10 200-219 152 152 152 268 268 268 F F F 11 220-239 150 150 150 263 263 263 F F F 12 240-259 148 148 148 258 258 258 F F F 13 260-279 148 148 148 260 260 260 F F F 14 280-299 153 153 153 269 269 269 F F F 15 300-319 150 150 150 265 265 265 F F F 16 320-339 150 150 150 264 264 264 F F F 17 340-359 151 151 151 266 266 266 F F F 18 360-379 148 148 148 261 261 261 F F F 19 380-399 150 150 150 263 263 263 F F F 20 400-419 153 153 153 269 269 269 F F F 21 420-439 153 153 153 269 269 269 F F F 22 440-459 144 144 144 255 255 255 F F F 23 460-479 152 152 152 268 268 268 F F F 24 480-499 150 150 150 265 265 265 F F F 25 500-519 148 148 148 261 261 261 F F F 26 520-539 151 151 151 266 266 266 F F F 27 540-559 151 151 151 266 266 266 F F F 28 560-579 153 153 153 269 269 269 F F F 29 580-599 151 151 151 266 266 266 F F F Max gen: 153, Max flush: 269 Begin loop 77 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.64713019Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.693206215Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.693851314Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.693905837Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.693924194Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.693947062Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822} {"msg":"Crucible 18c34f80-94d2-4dd2-b016-9e7b2b6679c9 has session id: bfeaec34-c889-41e6-8930-73c63a696bff","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.694477955Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"Upstairs opts: Upstairs UUID: 18c34f80-94d2-4dd2-b016-9e7b2b6679c9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.694518419Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"Crucible stats registered with UUID: 18c34f80-94d2-4dd2-b016-9e7b2b6679c9","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.694537337Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:03.694557714Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.694736982Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"io task","client":"0","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.694772447Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"io task","client":"1","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.694791335Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.69499295Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"task":"crutest"} {"msg":"The guest has requested activation with gen:154","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.695037014Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822} {"msg":"18c34f80-94d2-4dd2-b016-9e7b2b6679c9 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.695105326Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.695137132Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"client":"0","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.695155279Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"client":"1","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.695172187Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"client":"2","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.695188745Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"io task","client":"0","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.695206103Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.69522519Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"io task","client":"1","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"ds_connection connected from Ok(127.0.0.1:58888)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.695400778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"io task","client":"1","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"ds_connection connected from Ok(127.0.0.1:41370)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.695446013Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"io task","client":"0","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"ds_connection connected from Ok(127.0.0.1:44642)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.695614412Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.696581759Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"client":"2","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.696675278Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"client":"1","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.696839417Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"client":"0","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697256405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697285521Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"[0]R flush_numbers[0..12]: [256, 269, 249, 268, 269, 266, 256, 266, 266, 268, 268, 263]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697393727Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"downstairs","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"[0]R generation[0..12]: [146, 153, 140, 152, 153, 151, 146, 151, 151, 152, 152, 150]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697418224Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"downstairs","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697438542Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"downstairs","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"[1]R flush_numbers[0..12]: [256, 269, 249, 268, 269, 266, 256, 266, 266, 268, 268, 263]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697457959Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"downstairs","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"[1]R generation[0..12]: [146, 153, 140, 152, 153, 151, 146, 151, 151, 152, 152, 150]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697476797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"downstairs","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697496344Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"downstairs","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"[2]R flush_numbers[0..12]: [256, 269, 249, 268, 269, 266, 256, 266, 266, 268, 268, 263]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697515542Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"downstairs","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"[2]R generation[0..12]: [146, 153, 140, 152, 153, 151, 146, 151, 151, 152, 152, 150]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.69753447Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"downstairs","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697554517Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"downstairs","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"Max found gen is 154","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697575305Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"downstairs","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"Generation requested: 154 >= found:154","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697593172Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"downstairs","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"Next flush: 270","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.69760928Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"downstairs","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697625488Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"":"downstairs","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697642606Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-11-06T20:18:03.697657864Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"client":"0","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697725006Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"client":"1","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697743973Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"client":"2","":"downstairs","session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697761011Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697777119Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"18c34f80-94d2-4dd2-b016-9e7b2b6679c9 is now active with session: bfeaec34-c889-41e6-8930-73c63a696bff","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697795097Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697810585Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822,"session_id":"bfeaec34-c889-41e6-8930-73c63a696bff"} {"msg":"The guest has finished waiting for activation with:154","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697826763Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822} {"msg":"Activated sub_volume 18c34f80-94d2-4dd2-b016-9e7b2b6679c9","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:03.697841041Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 281 len 16384 data: 9 10 10 10 02/30 Read block 310 len 36864 03/30 Read block 435 len 24576 04/30 Write block 87 len 36864 data: 10 11 14 13 15 14 12 12 13 05/30 Read block 136 len 24576 06/30 Read block 400 len 24576 07/30 Write block 380 len 12288 data: 10 10 9 08/30 Write block 82 len 24576 data: 13 11 13 9 10 11 09/30 Write block 82 len 36864 data: 14 12 14 10 11 12 12 15 14 10/30 Read block 171 len 4096 11/30 Read block 336 len 4096 12/30 Flush 13/30 Write block 50 len 36864 data: 13 11 11 10 11 8 8 9 11 14/30 Write block 153 len 16384 data: 9 11 12 10 15/30 Flush 16/30 Write block 293 len 40960 data: 17 17 15 13 11 12 11 8 9 9 17/30 Read block 420 len 36864 18/30 Read block 541 len 8192 19/30 Read block 385 len 36864 20/30 Read block 387 len 40960 21/30 Read block 119 len 24576 22/30 Flush 23/30 Write block 394 len 4096 data: 13 24/30 Flush 25/30 Write block 451 len 24576 data: 21 17 18 19 18 17 26/30 Write block 348 len 24576 data: 9 9 11 9 9 12 27/30 Read block 387 len 40960 28/30 Write block 388 len 40960 data: 9 10 11 11 14 13 14 13 10 11 29/30 Write block 178 len 12288 data: 14 19 21 30/30 Read block 259 len 16384 ---------------------------------------------------------------- Crucible gen:154 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 9 Done Sent Done false 1002 Acked Read 6 Done Sent Done false 1003 Acked Write 9 Done Sent Done false 1004 Acked Read 6 Done Sent Done false 1005 Acked Read 6 Done Sent Done false 1006 Acked Write 3 Done Sent Done false 1007 Acked Write 6 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Read 1 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 9 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Read 9 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Read 6 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 1 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 6 Done Sent Done false 1025 Acked Write 6 Done Sent Done false 1026 Acked Read 10 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Write 3 Done Sent Done false 1029 Acked Read 4 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:03.717601931Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1822} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 256 256 256 F F F 1 020-039 153 153 153 269 269 269 F F F 2 040-059 154 140 154 271 249 271 F F F <--- 3 060-079 152 152 152 268 268 268 F F F 4 080-099 154 153 154 270 269 270 F F F <--- 5 100-119 151 151 151 266 266 266 F F F 6 120-139 146 146 146 256 256 256 F F F 7 140-159 154 151 154 271 266 271 F F F <--- 8 160-179 151 151 151 266 266 266 T F T <--- 9 180-199 152 152 152 268 268 268 T F T <--- 10 200-219 152 152 152 268 268 268 F F F 11 220-239 150 150 150 263 263 263 F F F 12 240-259 148 148 148 258 258 258 F F F 13 260-279 148 148 148 260 260 260 F F F 14 280-299 154 153 154 272 269 272 F T F <--- 15 300-319 154 150 154 272 265 272 F F F <--- 16 320-339 150 150 150 264 264 264 F F F 17 340-359 151 151 151 266 266 266 T F T <--- 18 360-379 148 148 148 261 261 261 F F F 19 380-399 154 150 154 273 263 273 T F T <--- 20 400-419 153 153 153 269 269 269 F F F 21 420-439 153 153 153 269 269 269 F F F 22 440-459 144 144 144 255 255 255 T F T <--- 23 460-479 152 152 152 268 268 268 F F F 24 480-499 150 150 150 265 265 265 F F F 25 500-519 148 148 148 261 261 261 F F F 26 520-539 151 151 151 266 266 266 F F F 27 540-559 151 151 151 266 266 266 F F F 28 560-579 153 153 153 269 269 269 F F F 29 580-599 151 151 151 266 266 266 F F F Max gen: 154, Max flush: 273 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 77 Verifying data now No change after verify Loop: 77 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 256 256 256 F F F 1 020-039 153 153 153 269 269 269 F F F 2 040-059 154 154 154 271 271 271 F F F 3 060-079 152 152 152 268 268 268 F F F 4 080-099 154 154 154 270 270 270 F F F 5 100-119 151 151 151 266 266 266 F F F 6 120-139 146 146 146 256 256 256 F F F 7 140-159 154 154 154 271 271 271 F F F 8 160-179 155 155 155 274 274 274 F F F 9 180-199 155 155 155 274 274 274 F F F 10 200-219 152 152 152 268 268 268 F F F 11 220-239 150 150 150 263 263 263 F F F 12 240-259 148 148 148 258 258 258 F F F 13 260-279 148 148 148 260 260 260 F F F 14 280-299 154 154 154 272 272 272 F F F 15 300-319 154 154 154 272 272 272 F F F 16 320-339 150 150 150 264 264 264 F F F 17 340-359 155 155 155 274 274 274 F F F 18 360-379 148 148 148 261 261 261 F F F 19 380-399 155 155 155 274 274 274 F F F 20 400-419 153 153 153 269 269 269 F F F 21 420-439 153 153 153 269 269 269 F F F 22 440-459 155 155 155 274 274 274 F F F 23 460-479 152 152 152 268 268 268 F F F 24 480-499 150 150 150 265 265 265 F F F 25 500-519 148 148 148 261 261 261 F F F 26 520-539 151 151 151 266 266 266 F F F 27 540-559 151 151 151 266 266 266 F F F 28 560-579 153 153 153 269 269 269 F F F 29 580-599 151 151 151 266 266 266 F F F Max gen: 155, Max flush: 274 Begin loop 78 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.116049449Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.161328134Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.161973473Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.162014728Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.162033926Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.162056283Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830} {"msg":"Crucible 3871b7d3-9054-4375-af97-aa0f15094e6b has session id: 78b7e7fa-7016-408e-a0af-53d9a5bcff4c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.162660357Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"Upstairs opts: Upstairs UUID: 3871b7d3-9054-4375-af97-aa0f15094e6b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.16271183Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"Crucible stats registered with UUID: 3871b7d3-9054-4375-af97-aa0f15094e6b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.162738357Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.162755205Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.162968738Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"io task","client":"0","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163018701Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"io task","client":"2","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163040879Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163210987Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"task":"crutest"} {"msg":"The guest has requested activation with gen:156","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163233825Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830} {"msg":"3871b7d3-9054-4375-af97-aa0f15094e6b active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163317544Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163344991Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"client":"0","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163381506Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"client":"1","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163415971Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"client":"2","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163446518Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"io task","client":"2","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163481343Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"io task","client":"0","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163504091Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"ds_connection connected from Ok(127.0.0.1:63926)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163535727Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"ds_connection connected from Ok(127.0.0.1:53364)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163568752Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"io task","client":"2","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"ds_connection connected from Ok(127.0.0.1:42329)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.163674279Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"io task","client":"0","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.164798787Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"client":"1","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.164834943Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"client":"0","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.165426738Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.165456685Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166022253Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"client":"2","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"[0]R flush_numbers[0..12]: [256, 269, 271, 268, 270, 266, 256, 271, 274, 274, 268, 263]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166529029Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"downstairs","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"[0]R generation[0..12]: [146, 153, 154, 152, 154, 151, 146, 154, 155, 155, 152, 150]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166559155Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"downstairs","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166583152Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"downstairs","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"[1]R flush_numbers[0..12]: [256, 269, 271, 268, 270, 266, 256, 271, 274, 274, 268, 263]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.1666031Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"downstairs","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"[1]R generation[0..12]: [146, 153, 154, 152, 154, 151, 146, 154, 155, 155, 152, 150]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166623298Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"downstairs","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166643095Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"downstairs","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"[2]R flush_numbers[0..12]: [256, 269, 271, 268, 270, 266, 256, 271, 274, 274, 268, 263]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166661873Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"downstairs","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"[2]R generation[0..12]: [146, 153, 154, 152, 154, 151, 146, 154, 155, 155, 152, 150]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.16668064Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"downstairs","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166700558Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","Wait for a query_work_queue command to finish before sending IO pid":1830,"":"downstairs","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"Max found gen is 156","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166725165Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"downstairs","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"Generation requested: 156 >= found:156","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166742732Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"downstairs","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"Next flush: 275","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.16675906Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"downstairs","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166775428Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"":"downstairs","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166793626Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166808954Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"client":"0","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166825862Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"client":"1","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.16684258Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"client":"2","":"downstairs","session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166861428Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166877785Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"3871b7d3-9054-4375-af97-aa0f15094e6b is now active with session: 78b7e7fa-7016-408e-a0af-53d9a5bcff4c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166895573Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166910691Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"The guest has finished waiting for activation with:156","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166926679Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830} {"msg":"Activated sub_volume 3871b7d3-9054-4375-af97-aa0f15094e6b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:08.166940818Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 140 len 16384 data: 10 12 12 10 02/30 Read block 508 len 4096 03/30 Write block 138 len 32768 data: 12 10 11 13 13 11 11 10 04/30 Read block 439 len 20480 05/30 Read block 246 len 8192 06/30 Read block 462 len 16384 07/30 Read block 49 len 12288 08/30 Write block 156 len 32768 data: 11 12 11 13 13 13 14 11 09/30 Read block 224 len 24576 10/30 Write block 221 len 28672 data: 11 10 13 10 7 7 8 11/30 Write block 179 len 16384 data: 20 22 22 21 12/30 Read block 393 len 4096 13/30 Write block 68 len 8192 data: 10 9 14/30 Write block 182 len 8192 data: 22 21 15/30 Read block 212 len 24576 16/30 Read block 114 len 20480 17/30 Read block 423 len 16384 18/30 Read block 78 len 32768 19/30 Flush 20/30 Write block 387 len 40960 data: 8 10 11 12 12 15 14 15 14 11 21/30 Read block 25 len 40960 22/30 Flush 23/30 Write block 449 len 36864 data: 18 22 22 18 19 20 19 18 16 24/30 Read block 28 len 28672 25/30 Write block 239 len 12288 data: 15 14 15 26/30 Flush 27/30 Write block 353 len 20480 data: 13 11 11 10 10 28/30 Write block 302 len 8192 data: 10 9 29/30 Read block 13 len 24576 30/30 Write block 138 len 28672 data: 13 11 12 14 14 12 12 ---------------------------------------------------------------- Crucible gen:156 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Read 5 Done Done Done false 1004 Acked Read 2 Done Done Done false 1005 Acked Read 4 Done Done Done false 1006 Acked Read 3 Done Done Done false 1007 Acked Write 8 Done Done Done false 1008 Acked Read 6 Done Done Done false 1009 Acked Write 7 Done Done Done false 1010 Acked Write 4 Done Done Sent false 1011 Acked Read 1 Done Done Sent false 1012 Acked Write 2 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 6 Done Done Sent false 1015 Acked Read 5 Done Done Sent false 1016 Acked Read 4 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Read 10 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Write 3 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Write 2 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 20 22 Done 29 29 10 68 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.187700462Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.1881908Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188233815Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.18826949Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188300227Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188330223Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188360519Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188385426Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188402823Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188418992Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188440389Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188482413Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188505481Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188521968Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188537677Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188553415Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188569042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188584611Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188601038Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188616876Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188632794Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188648483Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.18866441Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188680448Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188696526Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188712855Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188728782Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.18874488Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188760738Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188778346Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188794104Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188809782Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.18882565Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188841348Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188856926Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188872524Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188888182Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.18890393Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188919578Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188935167Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188950545Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188966132Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.18898195Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.188997699Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189013786Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189029895Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189045813Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.18906347Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189079308Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189094856Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189110704Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189126272Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.18914181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189157738Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189173406Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189189204Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189205073Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.18922088Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189236578Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189252427Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189268125Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189283983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.18929975Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189315468Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189333026Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189349174Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189365122Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.18938091Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189396588Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189412296Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189428044Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189443632Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189459141Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189474919Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189490706Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189506395Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189522062Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189537811Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189553358Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189568927Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189584595Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189602992Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.18961866Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189634528Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189650136Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189665724Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189681452Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189697051Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189712709Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189728276Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189743994Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189759812Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189775421Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189790979Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189806547Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189822095Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189837933Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189853611Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189871009Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189886807Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189902605Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189918463Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189933961Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189949699Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189965317Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189980995Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.189996733Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190012391Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190028069Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190043777Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190059625Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190075283Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190091061Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190106639Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190122347Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190139735Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190155353Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190171081Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190186689Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190202357Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190218035Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190233683Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190249251Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190265049Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190280767Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:08.190296225Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1830,"session_id":"78b7e7fa-7016-408e-a0af-53d9a5bcff4c"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 256 256 256 F F F 1 020-039 153 153 153 269 269 269 F F F 2 040-059 154 154 154 271 271 271 F F F 3 060-079 156 156 152 275 275 268 F F F <--- 4 080-099 154 154 154 270 270 270 F F F 5 100-119 151 151 151 266 266 266 F F F 6 120-139 156 156 146 275 275 256 T T T <--- 7 140-159 156 156 154 275 275 271 T T T <--- 8 160-179 156 156 155 275 275 274 F F T <--- 9 180-199 156 156 155 275 275 274 F F F <--- 10 200-219 152 152 152 268 268 268 F F F 11 220-239 156 156 150 277 277 263 F F T <--- 12 240-259 156 156 148 277 277 258 F F F <--- 13 260-279 148 148 148 260 260 260 F F F 14 280-299 154 154 154 272 272 272 F F F 15 300-319 154 154 154 272 272 272 T T F <--- 16 320-339 150 150 150 264 264 264 F F F 17 340-359 155 155 155 274 274 274 T T F <--- 18 360-379 148 148 148 261 261 261 F F F 19 380-399 156 156 155 276 276 274 F F F <--- 20 400-419 153 153 153 269 269 269 F F F 21 420-439 153 153 153 269 269 269 F F F 22 440-459 156 156 155 277 277 274 F F F <--- 23 460-479 152 152 152 268 268 268 F F F 24 480-499 150 150 150 265 265 265 F F F 25 500-519 148 148 148 261 261 261 F F F 26 520-539 151 151 151 266 266 266 F F F 27 540-559 151 151 151 266 266 266 F F F 28 560-579 153 153 153 269 269 269 F F F 29 580-599 151 151 151 266 266 266 F F F Max gen: 156, Max flush: 277 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 78 Verifying data now No change after verify Loop: 78 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 146 146 146 256 256 256 F F F 1 020-039 153 153 153 269 269 269 F F F 2 040-059 154 154 154 271 271 271 F F F 3 060-079 156 156 156 275 275 275 F F F 4 080-099 154 154 154 270 270 270 F F F 5 100-119 151 151 151 266 266 266 F F F 6 120-139 157 157 157 278 278 278 F F F 7 140-159 157 157 157 278 278 278 F F F 8 160-179 156 156 156 275 275 275 F F F 9 180-199 156 156 156 275 275 275 F F F 10 200-219 152 152 152 268 268 268 F F F 11 220-239 156 156 156 277 277 277 F F F 12 240-259 156 156 156 277 277 277 F F F 13 260-279 148 148 148 260 260 260 F F F 14 280-299 154 154 154 272 272 272 F F F 15 300-319 157 157 157 278 278 278 F F F 16 320-339 150 150 150 264 264 264 F F F 17 340-359 157 157 157 278 278 278 F F F 18 360-379 148 148 148 261 261 261 F F F 19 380-399 156 156 156 276 276 276 F F F 20 400-419 153 153 153 269 269 269 F F F 21 420-439 153 153 153 269 269 269 F F F 22 440-459 156 156 156 277 277 277 F F F 23 460-479 152 152 152 268 268 268 F F F 24 480-499 150 150 150 265 265 265 F F F 25 500-519 148 148 148 261 261 261 F F F 26 520-539 151 151 151 266 266 266 F F F 27 540-559 151 151 151 266 266 266 F F F 28 560-579 153 153 153 269 269 269 F F F 29 580-599 151 151 151 266 266 266 F F F Max gen: 157, Max flush: 278 Begin loop 79 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.588339974Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.633367741Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.633990022Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.634045465Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.634068792Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.634091739Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838} {"msg":"Crucible ba130cbd-8bc5-4a72-9f40-833204fc5db6 has session id: 3a009172-275b-4b3d-9cac-5dc2d95274cb","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.634698693Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Upstairs opts: Upstairs UUID: ba130cbd-8bc5-4a72-9f40-833204fc5db6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.634740348Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Crucible stats registered with UUID: ba130cbd-8bc5-4a72-9f40-833204fc5db6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.634759355Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:12.634775243Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.634986747Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"io task","client":"0","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635027092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635056468Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"io task","client":"1","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635242885Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"task":"crutest"} {"msg":"The guest has requested activation with gen:158","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635265451Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838} {"msg":"ba130cbd-8bc5-4a72-9f40-833204fc5db6 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635331173Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635351731Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"client":"0","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635369218Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"client":"1","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635386326Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"client":"2","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635402884Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635424031Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"io task","client":"0","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635445469Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"io task","client":"1","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"ds_connection connected from Ok(127.0.0.1:42979)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635555765Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"ds_connection connected from Ok(127.0.0.1:55691)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635656902Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"io task","client":"1","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"ds_connection connected from Ok(127.0.0.1:65471)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.635697957Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"io task","client":"0","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.636653157Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"client":"1","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.636737746Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"client":"0","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.637240573Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.637335401Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638070418Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"client":"2","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"[0]R flush_numbers[0..12]: [256, 269, 271, 275, 270, 266, 278, 278, 275, 275, 268, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638548938Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"downstairs","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"[0]R generation[0..12]: [146, 153, 154, 156, 154, 151, 157, 157, 156, 156, 152, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638574545Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"downstairs","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638594992Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"downstairs","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"[1]R flush_numbers[0..12]: [256, 269, 271, 275, 270, 266, 278, 278, 275, 275, 268, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.63861417Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"downstairs","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"[1]R generation[0..12]: [146, 153, 154, 156, 154, 151, 157, 157, 156, 156, 152, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638641086Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"downstairs","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638661424Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"downstairs","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"[2]R flush_numbers[0..12]: [256, 269, 271, 275, 270, 266, 278, 278, 275, 275, 268, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638680271Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"downstairs","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"[2]R generation[0..12]: [146, 153, 154, 156, 154, 151, 157, 157, 156, 156, 152, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638700389Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"downstairs","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638720396Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO 3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Max found gen is 158","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638743373Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"downstairs","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Generation requested: 158 >= found:158","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638760371Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"downstairs","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Next flush: 279","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638776609Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"downstairs","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638792537Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"":"downstairs","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638809575Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638824763Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"client":"0","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638841961Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"client":"1","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638858759Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"client":"2","":"downstairs","session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638875717Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638891765Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"ba130cbd-8bc5-4a72-9f40-833204fc5db6 is now active with session: 3a009172-275b-4b3d-9cac-5dc2d95274cb","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638912062Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.63892739Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838,"session_id":"3a009172-275b-4b3d-9cac-5dc2d95274cb"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 The guest has finished waiting for activation with:158","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638946278Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838} {"msg":"Activated sub_volume ba130cbd-8bc5-4a72-9f40-833204fc5db6","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:12.638960646Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 202 len 20480 data: 15 15 15 17 13 02/30 Read block 408 len 28672 03/30 Write block 395 len 36864 data: 15 12 12 11 12 12 14 13 13 04/30 Read block 329 len 16384 05/30 Read block 374 len 12288 06/30 Write block 215 len 12288 data: 7 8 8 07/30 Read block 67 len 20480 08/30 Read block 394 len 40960 09/30 Write block 18 len 12288 data: 3 6 11 10/30 Write block 378 len 40960 data: 8 9 11 11 10 10 10 13 11 9 11/30 Read block 294 len 32768 12/30 Read block 178 len 40960 13/30 Read block 495 len 16384 14/30 Write block 76 len 32768 data: 11 12 16 14 13 13 15 13 15/30 Flush 16/30 Write block 425 len 40960 data: 13 13 11 13 12 13 10 8 8 12 17/30 Read block 221 len 4096 18/30 Read block 432 len 32768 19/30 Write block 76 len 16384 data: 12 13 17 15 20/30 Write block 257 len 28672 data: 10 12 6 6 6 6 5 21/30 Write block 95 len 4096 data: 14 22/30 Read block 501 len 32768 23/30 Write block 312 len 40960 data: 15 14 15 12 10 10 10 11 10 12 24/30 Flush 25/30 Write block 47 len 32768 data: 17 14 13 14 12 12 11 12 26/30 Write block 525 len 24576 data: 11 10 9 10 9 10 27/30 Write block 245 len 40960 data: 19 17 14 13 13 13 11 9 10 13 28/30 Read block 200 len 36864 29/30 Write block 304 len 20480 data: 9 11 14 16 15 30/30 Write block 506 len 40960 data: 12 8 10 9 12 11 12 11 11 12 ---------------------------------------------------------------- Crucible gen:158 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Write 9 Done Done Done false 1003 Acked Read 4 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Write 3 Done Done Done false 1006 Acked Read 5 Done Done Done false 1007 Acked Read 10 Done Done Sent false 1008 Acked Write 3 Done Done Sent false 1009 Acked Write 10 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Read 10 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Write 8 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Write 1 Done Done Sent false 1021 Acked Read 8 Done Done Sent false 1022 Acked Write 10 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 8 Done Done Sent false 1025 Acked Write 6 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Write 5 Done Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 2 23 26 Done 29 28 7 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:12.659984817Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1838} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 158 158 146 279 279 256 F F F <--- 1 020-039 158 158 153 279 279 269 F F F <--- 2 040-059 154 154 154 271 271 271 T T F <--- 3 060-079 158 158 156 280 280 275 F F F <--- 4 080-099 158 158 154 280 280 270 F F F <--- 5 100-119 151 151 151 266 266 266 F F F 6 120-139 157 157 157 278 278 278 F F F 7 140-159 157 157 157 278 278 278 F F F 8 160-179 156 156 156 275 275 275 F F F 9 180-199 156 156 156 275 275 275 F F F 10 200-219 158 158 152 279 279 268 F F T <--- 11 220-239 156 156 156 277 277 277 F F F 12 240-259 158 158 156 280 280 277 T T F <--- 13 260-279 158 158 148 280 280 260 F F F <--- 14 280-299 154 154 154 272 272 272 F F F 15 300-319 158 158 157 280 280 278 T T F <--- 16 320-339 158 158 150 280 280 264 F F F <--- 17 340-359 157 157 157 278 278 278 F F F 18 360-379 158 158 148 279 279 261 F F F <--- 19 380-399 158 158 156 279 279 276 F F T <--- 20 400-419 158 158 153 279 279 269 F F T <--- 21 420-439 158 158 153 280 280 269 F F F <--- 22 440-459 156 156 156 277 277 277 F F F 23 460-479 152 152 152 268 268 268 F F F 24 480-499 150 150 150 265 265 265 F F F 25 500-519 148 148 148 261 261 261 T T F <--- 26 520-539 151 151 151 266 266 266 T T F <--- 27 540-559 151 151 151 266 266 266 F F F 28 560-579 153 153 153 269 269 269 F F F 29 580-599 151 151 151 266 266 266 F F F Max gen: 158, Max flush: 280 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 79 Verifying data now No change after verify Loop: 79 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 158 158 158 279 279 279 F F F 1 020-039 158 158 158 279 279 279 F F F 2 040-059 159 159 159 281 281 281 F F F 3 060-079 158 158 158 280 280 280 F F F 4 080-099 158 158 158 280 280 280 F F F 5 100-119 151 151 151 266 266 266 F F F 6 120-139 157 157 157 278 278 278 F F F 7 140-159 157 157 157 278 278 278 F F F 8 160-179 156 156 156 275 275 275 F F F 9 180-199 156 156 156 275 275 275 F F F 10 200-219 158 158 158 279 279 279 F F F 11 220-239 156 156 156 277 277 277 F F F 12 240-259 159 159 159 281 281 281 F F F 13 260-279 158 158 158 280 280 280 F F F 14 280-299 154 154 154 272 272 272 F F F 15 300-319 159 159 159 281 281 281 F F F 16 320-339 158 158 158 280 280 280 F F F 17 340-359 157 157 157 278 278 278 F F F 18 360-379 158 158 158 279 279 279 F F F 19 380-399 158 158 158 279 279 279 F F F 20 400-419 158 158 158 279 279 279 F F F 21 420-439 158 158 158 280 280 280 F F F 22 440-459 156 156 156 277 277 277 F F F 23 460-479 152 152 152 268 268 268 F F F 24 480-499 150 150 150 265 265 265 F F F 25 500-519 159 159 159 281 281 281 F F F 26 520-539 159 159 159 281 281 281 F F F 27 540-559 151 151 151 266 266 266 F F F 28 560-579 153 153 153 269 269 269 F F F 29 580-599 151 151 151 266 266 266 F F F Max gen: 159, Max flush: 281 Begin loop 80 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.077010555Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.123810759Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.124426181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.124502452Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.124538967Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.124584641Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846} {"msg":"Crucible a0f9f183-339b-4633-8e6d-62c6d5d2f514 has session id: 9cc50992-842f-4f62-841b-6560982a78ba","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125136281Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"Upstairs opts: Upstairs UUID: a0f9f183-339b-4633-8e6d-62c6d5d2f514, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125205773Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"Crucible stats registered with UUID: a0f9f183-339b-4633-8e6d-62c6d5d2f514","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125245868Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:17.125282183Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125438274Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125476399Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"io task","client":"1","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125496356Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"io task","client":"0","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125655946Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"task":"crutest"} {"msg":"The guest has requested activation with gen:160","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125694271Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846} {"msg":"a0f9f183-339b-4633-8e6d-62c6d5d2f514 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125749284Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125773371Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"client":"0","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125792349Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"client":"1","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125809337Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"client":"2","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125825825Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125843303Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"io task","client":"0","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125871569Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"io task","client":"1","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"ds_connection connected from Ok(127.0.0.1:39591)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125903865Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"ds_connection connected from Ok(127.0.0.1:65409)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.125952399Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"io task","client":"0","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"ds_connection connected from Ok(127.0.0.1:65174)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.126054466Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"io task","client":"1","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.127140679Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"client":"0","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.127185554Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"client":"2","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.127516662Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"client":"1","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.127722666Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.127806235Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"[0]R flush_numbers[0..12]: [279, 279, 281, 280, 280, 266, 278, 278, 275, 275, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128045785Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"downstairs","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"[0]R generation[0..12]: [158, 158, 159, 158, 158, 151, 157, 157, 156, 156, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128080351Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"downstairs","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128102218Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"downstairs","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"[1]R flush_numbers[0..12]: [279, 279, 281, 280, 280, 266, 278, 278, 275, 275, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128122125Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"downstairs","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"[1]R generation[0..12]: [158, 158, 159, 158, 158, 151, 157, 157, 156, 156, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128141573Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"downstairs","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128161351Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"downstairs","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"[2]R flush_numbers[0..12]: [279, 279, 281, 280, 280, 266, 278, 278, 275, 275, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128180578Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"downstairs","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"[2]R generation[0..12]: [158, 158, 159, 158, 158, 151, 157, 157, 156, 156, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128199486Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"downstairs","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128219023Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"downstairs","":"downstairs","session_id"Wait for a query_work_queue command to finish before sending IO :"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"Max found gen is 160","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.12824508Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"downstairs","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"Generation requested: 160 >= found:160","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128267507Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"downstairs","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"Next flush: 282","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128284065Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"downstairs","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128300283Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"":"downstairs","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128317541Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128332699Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"client":"0","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128349697Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"client":"1","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128366595Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"client":"2","":"downstairs","session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128387632Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.1284042Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"a0f9f183-339b-4633-8e6d-62c6d5d2f514 is now active with session: 9cc50992-842f-4f62-841b-6560982a78ba","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128422478Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128437806Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846,"session_id":"9cc50992-842f-4f62-841b-6560982a78ba"} {"msg":"The guest has finished waiting for activation with:160","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128453684Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846} {"msg":"Activated sub_volume a0f9f183-339b-4633-8e6d-62c6d5d2f514","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:17.128467422Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 128 len 24576 data: 14 17 15 14 19 19 02/30 Flush 03/30 Write block 20 len 16384 data: 12 11 16 16 04/30 Read block 520 len 40960 05/30 Write block 367 len 28672 data: 8 9 8 7 8 7 6 06/30 Write block 31 len 4096 data: 11 07/30 Read block 581 len 12288 08/30 Read block 375 len 40960 09/30 Write block 544 len 20480 data: 16 14 11 12 13 10/30 Read block 41 len 16384 11/30 Write block 274 len 36864 data: 14 15 12 14 13 12 9 10 11 12/30 Read block 292 len 32768 13/30 Write block 370 len 8192 data: 8 9 14/30 Write block 420 len 20480 data: 16 12 14 12 13 15/30 Read block 280 len 32768 16/30 Write block 372 len 36864 data: 8 7 7 6 6 8 9 10 12 17/30 Read block 526 len 12288 18/30 Read block 156 len 8192 19/30 Read block 116 len 32768 20/30 Flush 21/30 Write block 402 len 28672 data: 14 14 13 12 10 10 11 22/30 Write block 285 len 4096 data: 9 23/30 Read block 131 len 16384 24/30 Write block 89 len 32768 data: 16 15 16 15 13 13 15 13 25/30 Write block 512 len 28672 data: 13 12 12 13 12 13 12 26/30 Read block 540 len 4096 27/30 Write block 269 len 16384 data: 12 12 14 14 28/30 Read block 190 len 20480 29/30 Read block 237 len 28672 30/30 Read block 37 len 4096 ---------------------------------------------------------------- Crucible gen:160 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Sent Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 4 Done Sent Done false 1003 Acked Read 10 Done Sent Done false 1004 Acked Write 7 Done Sent Done false 1005 Acked Write 1 Done Sent Done false 1006 Acked Read 3 Done Sent Done false 1007 Acked Read 10 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Write 9 Done Sent Done false 1011 Acked Read 8 Done Sent Done false 1012 Acked Write 2 Done Sent Done false 1013 Acked Write 5 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Read 3 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Read 8 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Write 1 Done Sent Done false 1022 Acked Read 4 Done Sent Done false 1023 Acked Write 8 Done Sent Done false 1024 Acked Write 7 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 1 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:17.148793391Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1846} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 158 158 158 279 279 279 F F F 1 020-039 160 158 160 283 279 283 F F F <--- 2 040-059 159 159 159 281 281 281 F F F 3 060-079 158 158 158 280 280 280 F F F 4 080-099 158 158 158 280 280 280 T F T <--- 5 100-119 151 151 151 266 266 266 F F F 6 120-139 160 157 160 282 278 282 F F F <--- 7 140-159 157 157 157 278 278 278 F F F 8 160-179 156 156 156 275 275 275 F F F 9 180-199 156 156 156 275 275 275 F F F 10 200-219 158 158 158 279 279 279 F F F 11 220-239 156 156 156 277 277 277 F F F 12 240-259 159 159 159 281 281 281 F F F 13 260-279 160 158 160 283 280 283 T F T <--- 14 280-299 160 154 160 283 272 283 T F T <--- 15 300-319 159 159 159 281 281 281 F F F 16 320-339 158 158 158 280 280 280 F F F 17 340-359 157 157 157 278 278 278 F F F 18 360-379 160 158 160 283 279 283 F F F <--- 19 380-399 160 158 160 283 279 283 F F F <--- 20 400-419 158 158 158 279 279 279 T F T <--- 21 420-439 160 158 160 283 280 283 F F F <--- 22 440-459 156 156 156 277 277 277 F F F 23 460-479 152 152 152 268 268 268 F F F 24 480-499 150 150 150 265 265 265 F F F 25 500-519 159 159 159 281 281 281 T F T <--- 26 520-539 159 159 159 281 281 281 F F F 27 540-559 160 151 160 283 266 283 F F F <--- 28 560-579 153 153 153 269 269 269 F F F 29 580-599 151 151 151 266 266 266 F F F Max gen: 160, Max flush: 283 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 80 Verifying data now No change after verify Loop: 80 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 158 158 158 279 279 279 F F F 1 020-039 160 160 160 283 283 283 F F F 2 040-059 159 159 159 281 281 281 F F F 3 060-079 158 158 158 280 280 280 F F F 4 080-099 161 161 161 284 284 284 F F F 5 100-119 151 151 151 266 266 266 F F F 6 120-139 160 160 160 282 282 282 F F F 7 140-159 157 157 157 278 278 278 F F F 8 160-179 156 156 156 275 275 275 F F F 9 180-199 156 156 156 275 275 275 F F F 10 200-219 158 158 158 279 279 279 F F F 11 220-239 156 156 156 277 277 277 F F F 12 240-259 159 159 159 281 281 281 F F F 13 260-279 161 161 161 284 284 284 F F F 14 280-299 161 161 161 284 284 284 F F F 15 300-319 159 159 159 281 281 281 F F F 16 320-339 158 158 158 280 280 280 F F F 17 340-359 157 157 157 278 278 278 F F F 18 360-379 160 160 160 283 283 283 F F F 19 380-399 160 160 160 283 283 283 F F F 20 400-419 161 161 161 284 284 284 F F F 21 420-439 160 160 160 283 283 283 F F F 22 440-459 156 156 156 277 277 277 F F F 23 460-479 152 152 152 268 268 268 F F F 24 480-499 150 150 150 265 265 265 F F F 25 500-519 161 161 161 284 284 284 F F F 26 520-539 159 159 159 281 281 281 F F F 27 540-559 160 160 160 283 283 283 F F F 28 560-579 153 153 153 269 269 269 F F F 29 580-599 151 151 151 266 266 266 F F F Max gen: 161, Max flush: 284 Begin loop 81 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.546499163Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:21.59073784Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.59112973Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.622784792Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.623327983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.623375857Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.623392675Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.623414823Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854} {"msg":"Crucible e41fb437-4e94-4d82-8382-e24ffb9c702e has session id: 73ce15fa-a699-4c35-880a-8eac6f86832f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624004718Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"Upstairs opts: Upstairs UUID: e41fb437-4e94-4d82-8382-e24ffb9c702e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624048123Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"Crucible stats registered with UUID: e41fb437-4e94-4d82-8382-e24ffb9c702e","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.62406808Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:21.624084638Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.62431113Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624351074Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"io task","client":"2","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624370882Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"io task","client":"0","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624529312Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"task":"crutest"} {"msg":"The guest has requested activation with gen:162","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624551579Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854} {"msg":"e41fb437-4e94-4d82-8382-e24ffb9c702e active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624619081Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624640758Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"client":"0","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624658586Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"client":"1","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624675504Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"client":"2","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624692092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"io task","client":"2","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624711549Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"io task","client":"0","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624740656Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"ds_connection connected from Ok(127.0.0.1:51498)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624775621Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"io task","client":"0","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"ds_connection connected from Ok(127.0.0.1:54517)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624810787Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"io task","client":"2","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"ds_connection connected from Ok(127.0.0.1:42795)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.624845092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.626253465Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"client":"1","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.62629436Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"client":"2","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.626361731Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"client":"0","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.62684816Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.626877476Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"[0]R flush_numbers[0..12]: [279, 283, 281, 280, 284, 266, 282, 278, 275, 275, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.626899613Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"downstairs","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"[0]R generation[0..12]: [158, 160, 159, 158, 161, 151, 160, 157, 156, 156, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.626919191Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"downstairs","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.626939139Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"downstairs","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"[1]R flush_numbers[0..12]: [279, 283, 281, 280, 284, 266, 282, 278, 275, 275, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.626958376Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"downstairs","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"[1]R generation[0..12]: [158, 160, 159, 158, 161, 151, 160, 157, 156, 156, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.626977174Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"downstairs","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.626996791Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"downstairs","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"[2]R flush_numbers[0..12]: [279, 283, 281, 280, 284, 266, 282, 278, 275, 275, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627016379Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"downstairs","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"[2]R generation[0..12]: [158, 160, 159, 158, 161, 151, 160, 157, 156, 156, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627035346Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"downstairs","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627054884Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"downstairs","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"Max found gen is 162","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627071012Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"downstairs","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"Generation requested: 162 >= found:162","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.62708778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"downstairs","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"Next flush: 285"Wait for a query_work_queue command to finish before sending IO ,"v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627110107Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"downstairs","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627126535Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"":"downstairs","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627143393Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627160601Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"client":"0","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627177618Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"client":"1","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627194226Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"client":"2","":"downstairs","session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627211284Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627227492Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"e41fb437-4e94-4d82-8382-e24ffb9c702e is now active with session: 73ce15fa-a699-4c35-880a-8eac6f86832f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.62724565Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627261008Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854,"session_id":"73ce15fa-a699-4c35-880a-8eac6f86832f"} {"msg":"The guest has finished waiting for activation with:162","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627277276Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854} {"msg":"Activated sub_volume e41fb437-4e94-4d82-8382-e24ffb9c702e","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:21.627291254Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 190 len 32768 data: 10 12 12 11 11 9 11 15 02/30 Read block 310 len 12288 03/30 Read block 403 len 8192 04/30 Write block 289 len 12288 data: 17 19 16 05/30 Flush 06/30 Flush 07/30 Write block 389 len 28672 data: 12 13 13 16 15 16 16 08/30 Read block 338 len 12288 09/30 Flush 10/30 Write block 425 len 40960 data: 14 14 12 14 13 14 11 9 9 13 11/30 Read block 420 len 40960 12/30 Write block 265 len 32768 data: 8 13 15 12 13 13 15 15 13/30 Write block 590 len 40960 data: 19 16 16 13 12 12 12 7 5 3 14/30 Write block 380 len 24576 data: 13 12 11 11 11 14 15/30 Read block 277 len 24576 16/30 Write block 387 len 8192 data: 10 11 17/30 Write block 499 len 20480 data: 11 9 11 11 11 18/30 Read block 154 len 8192 19/30 Read block 13 len 4096 20/30 Write block 365 len 20480 data: 9 9 9 10 9 21/30 Write block 283 len 36864 data: 11 11 10 10 11 9 18 20 17 22/30 Flush 23/30 Write block 25 len 12288 data: 17 16 16 24/30 Read block 142 len 20480 25/30 Read block 543 len 8192 26/30 Read block 190 len 24576 27/30 Write block 108 len 12288 data: 14 14 17 28/30 Write block 122 len 36864 data: 13 15 15 15 14 14 15 18 16 29/30 Read block 57 len 36864 30/30 Write block 161 len 12288 data: 14 15 12 ---------------------------------------------------------------- Crucible gen:162 GIO:true work queues: Upstairs:0 downstairs:24 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1006 Acked Write 7 Sent Done Done false 1007 Acked Read 3 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Read 10 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Write 10 Sent Done Done false 1013 Acked Write 6 Sent Done Done false 1014 Acked Read 6 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Write 5 Sent Done Done false 1017 Acked Read 2 Sent Done Done false 1018 Acked Read 1 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 3 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Read 2 Sent Done Done false 1025 Acked Read 6 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Write 9 Sent Done Done false 1028 Acked Read 9 Sent Done Done false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 1 1 26 Done 0 23 23 46 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1005)) Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: 1005 1004 1003 1002 1001 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:21.647854384Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1854} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 158 158 158 279 279 279 F F F 1 020-039 160 160 160 283 283 283 F T T <--- 2 040-059 159 159 159 281 281 281 F F F 3 060-079 158 158 158 280 280 280 F F F 4 080-099 161 161 161 284 284 284 F F F 5 100-119 151 151 151 266 266 266 F T T <--- 6 120-139 160 160 160 282 282 282 F T T <--- 7 140-159 157 157 157 278 278 278 F F F 8 160-179 156 156 156 275 275 275 F T T <--- 9 180-199 162 162 162 285 285 285 F F F 10 200-219 158 158 158 279 279 279 F F F 11 220-239 156 156 156 277 277 277 F F F 12 240-259 159 159 159 281 281 281 F F F 13 260-279 161 162 162 284 288 288 F F F <--- 14 280-299 162 162 162 285 288 288 F F F <--- 15 300-319 159 159 159 281 281 281 F F F 16 320-339 158 158 158 280 280 280 F F F 17 340-359 157 157 157 278 278 278 F F F 18 360-379 160 162 162 283 288 288 F F F <--- 19 380-399 160 162 162 283 288 288 F F F <--- 20 400-419 161 161 161 284 284 284 F F F 21 420-439 160 162 162 283 288 288 F F F <--- 22 440-459 156 156 156 277 277 277 F F F 23 460-479 152 152 152 268 268 268 F F F 24 480-499 150 162 162 265 288 288 F F F <--- 25 500-519 161 162 162 284 288 288 F F F <--- 26 520-539 159 159 159 281 281 281 F F F 27 540-559 160 160 160 283 283 283 F F F 28 560-579 153 153 153 269 269 269 F F F 29 580-599 151 162 162 266 288 288 F F F <--- Max gen: 162, Max flush: 288 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 81 Verifying data now No change after verify Loop: 81 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 158 158 158 279 279 279 F F F 1 020-039 163 163 163 289 289 289 F F F 2 040-059 159 159 159 281 281 281 F F F 3 060-079 158 158 158 280 280 280 F F F 4 080-099 161 161 161 284 284 284 F F F 5 100-119 163 163 163 289 289 289 F F F 6 120-139 163 163 163 289 289 289 F F F 7 140-159 157 157 157 278 278 278 F F F 8 160-179 163 163 163 289 289 289 F F F 9 180-199 162 162 162 285 285 285 F F F 10 200-219 158 158 158 279 279 279 F F F 11 220-239 156 156 156 277 277 277 F F F 12 240-259 159 159 159 281 281 281 F F F 13 260-279 162 162 162 288 288 288 F F F 14 280-299 162 162 162 288 288 288 F F F 15 300-319 159 159 159 281 281 281 F F F 16 320-339 158 158 158 280 280 280 F F F 17 340-359 157 157 157 278 278 278 F F F 18 360-379 162 162 162 288 288 288 F F F 19 380-399 162 162 162 288 288 288 F F F 20 400-419 161 161 161 284 284 284 F F F 21 420-439 162 162 162 288 288 288 F F F 22 440-459 156 156 156 277 277 277 F F F 23 460-479 152 152 152 268 268 268 F F F 24 480-499 162 162 162 288 288 288 F F F 25 500-519 162 162 162 288 288 288 F F F 26 520-539 159 159 159 281 281 281 F F F 27 540-559 160 160 160 283 283 283 F F F 28 560-579 153 153 153 269 269 269 F F F 29 580-599 162 162 162 288 288 288 F F F Max gen: 163, Max flush: 289 Begin loop 82 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.080014605Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.125733405Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.126345437Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.126393321Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.126409709Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.126432146Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862} {"msg":"Crucible adaacf9f-eb5e-40eb-8458-a1268c74c519 has session id: 521437b7-2d14-4e26-82e3-251b7ca21e3c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.126970079Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"Upstairs opts: Upstairs UUID: adaacf9f-eb5e-40eb-8458-a1268c74c519, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.127026432Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"Crucible stats registered with UUID: adaacf9f-eb5e-40eb-8458-a1268c74c519","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.127059178Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:26.127079975Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.127361599Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.127418972Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"io task","client":"1","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.127456508Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"io task","client":"2","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.127710915Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"task":"crutest"} {"msg":"The guest has requested activation with gen:164","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.127841619Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862} {"msg":"adaacf9f-eb5e-40eb-8458-a1268c74c519 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.127874745Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.127903231Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"client":"0","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.127936887Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"client":"1","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.127967233Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"client":"2","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.128031545Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"io task","client":"2","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.1280679Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.128102146Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"io task","client":"1","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"ds_connection connected from Ok(127.0.0.1:34970)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.128134602Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"io task","client":"2","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"ds_connection connected from Ok(127.0.0.1:55043)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.128172437Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"io task","client":"1","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"ds_connection connected from Ok(127.0.0.1:58321)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.128209433Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.129437708Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"client":"2","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.1295012Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"client":"0","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.129540665Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"client":"1","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130039302Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130066059Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"[0]R flush_numbers[0..12]: [279, 289, 281, 280, 284, 289, 289, 278, 289, 285, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130084526Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"downstairs","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"[0]R generation[0..12]: [158, 163, 159, 158, 161, 163, 163, 157, 163, 162, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130103974Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"downstairs","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130123851Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"downstairs","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"[1]R flush_numbers[0..12]: [279, 289, 281, 280, 284, 289, 289, 278, 289, 285, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130142929Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"downstairs","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"[1]R generation[0..12]: [158, 163, 159, 158, 161, 163, 163, 157, 163, 162, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130162076Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"downstairs","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130182044Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"downstairs","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"[2]R flush_numbers[0..12]: [279, 289, 281, 280, 284, 289, 289, 278, 289, 285, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130200832Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"downstairs","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"[2]R generation[0..12]: [158, 163, 159, 158, 161, 163, 163, 157, 163, 162, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130222999Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"downstairs","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130242576Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"downstairs","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"Max found gen is 164","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130258814Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"downstairs","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"Generation requested: 164 >= found:164","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130275912Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"downstairs","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"Next flush: 290","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.13029188Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"downstairs","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"All extents match","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2025-11-06T20:18:26.130314267Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"":"downstairs","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130331395Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130346753Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"client":"0","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130363561Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"client":"1","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130380529Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"client":"2","":"downstairs","session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130397697Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130413615Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"adaacf9f-eb5e-40eb-8458-a1268c74c519 is now active with session: 521437b7-2d14-4e26-82e3-251b7ca21e3c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130431773Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130446941Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862,"session_id":"521437b7-2d14-4e26-82e3-251b7ca21e3c"} {"msg":"The guest has finished waiting for activation with:164","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130462909Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862} {"msg":"Activated sub_volume adaacf9f-eb5e-40eb-8458-a1268c74c519","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:26.130476787Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 70 len 20480 data: 8 8 7 5 8 02/30 Read block 331 len 20480 03/30 Flush 04/30 Write block 571 len 16384 data: 11 9 10 8 05/30 Read block 167 len 16384 06/30 Read block 533 len 40960 07/30 Read block 138 len 12288 08/30 Write block 342 len 36864 data: 8 8 9 8 9 9 10 10 12 09/30 Write block 64 len 16384 data: 10 10 9 8 10/30 Read block 303 len 40960 11/30 Write block 65 len 36864 data: 11 10 9 11 10 9 9 8 6 12/30 Write block 587 len 36864 data: 13 17 20 20 17 17 14 13 13 13/30 Write block 41 len 8192 data: 22 19 14/30 Write block 329 len 12288 data: 9 9 9 15/30 Read block 263 len 32768 16/30 Read block 134 len 32768 17/30 Read block 88 len 8192 18/30 Write block 472 len 32768 data: 16 16 13 14 12 11 14 11 19/30 Read block 546 len 24576 20/30 Write block 43 len 16384 data: 16 16 17 17 21/30 Write block 534 len 24576 data: 13 14 15 14 15 16 22/30 Read block 286 len 28672 23/30 Write block 185 len 12288 data: 16 16 15 24/30 Write block 66 len 28672 data: 11 10 12 11 10 10 9 25/30 Read block 472 len 28672 26/30 Read block 276 len 36864 27/30 Read block 463 len 36864 28/30 Write block 264 len 24576 data: 6 9 14 16 13 14 29/30 Read block 32 len 16384 30/30 Write block 405 len 20480 data: 13 11 11 12 13 ---------------------------------------------------------------- Crucible gen:164 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Read 5 Done Sent Done false 1002 Acked Flush 0 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Read 4 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Read 3 Done Sent Done false 1007 Acked Write 9 Done Sent Done false 1008 Acked Write 4 Done Sent Done false 1009 Acked Read 10 Done Sent Done false 1010 Acked Write 9 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Write 2 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Write 8 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Write 6 Done Sent Done false 1021 Acked Read 7 Done Sent Done false 1022 Acked Write 3 Done Sent Done false 1023 Acked Write 7 Done Sent Done false 1024 Acked Read 7 Done Sent Done false 1025 Acked Read 9 Done Sent Done false 1026 Acked Read 9 Done Sent Done false 1027 Acked Write 6 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 29 1 31 Done 29 1 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) None Some(JobId(1002)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:26.149832938Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1862} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 158 158 158 279 279 279 F F F 1 020-039 163 163 163 289 289 289 F F F 2 040-059 159 159 159 281 281 281 T F T <--- 3 060-079 164 158 164 290 280 290 T T T <--- 4 080-099 161 161 161 284 284 284 F F F 5 100-119 163 163 163 289 289 289 F F F 6 120-139 163 163 163 289 289 289 F F F 7 140-159 157 157 157 278 278 278 F F F 8 160-179 163 163 163 289 289 289 F F F 9 180-199 162 162 162 285 285 285 T F T <--- 10 200-219 158 158 158 279 279 279 F F F 11 220-239 156 156 156 277 277 277 F F F 12 240-259 159 159 159 281 281 281 F F F 13 260-279 162 162 162 288 288 288 T F T <--- 14 280-299 162 162 162 288 288 288 F F F 15 300-319 159 159 159 281 281 281 F F F 16 320-339 158 158 158 280 280 280 T F T <--- 17 340-359 157 157 157 278 278 278 T F T <--- 18 360-379 162 162 162 288 288 288 F F F 19 380-399 162 162 162 288 288 288 F F F 20 400-419 161 161 161 284 284 284 T F T <--- 21 420-439 162 162 162 288 288 288 F F F 22 440-459 156 156 156 277 277 277 F F F 23 460-479 152 152 152 268 268 268 T F T <--- 24 480-499 162 162 162 288 288 288 F F F 25 500-519 162 162 162 288 288 288 F F F 26 520-539 159 159 159 281 281 281 T F T <--- 27 540-559 160 160 160 283 283 283 F F F 28 560-579 153 153 153 269 269 269 T F T <--- 29 580-599 162 162 162 288 288 288 T F T <--- Max gen: 164, Max flush: 290 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 82 Verifying data now No change after verify Loop: 82 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 158 158 158 279 279 279 F F F 1 020-039 163 163 163 289 289 289 F F F 2 040-059 165 165 165 291 291 291 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 161 161 161 284 284 284 F F F 5 100-119 163 163 163 289 289 289 F F F 6 120-139 163 163 163 289 289 289 F F F 7 140-159 157 157 157 278 278 278 F F F 8 160-179 163 163 163 289 289 289 F F F 9 180-199 165 165 165 291 291 291 F F F 10 200-219 158 158 158 279 279 279 F F F 11 220-239 156 156 156 277 277 277 F F F 12 240-259 159 159 159 281 281 281 F F F 13 260-279 165 165 165 291 291 291 F F F 14 280-299 162 162 162 288 288 288 F F F 15 300-319 159 159 159 281 281 281 F F F 16 320-339 165 165 165 291 291 291 F F F 17 340-359 165 165 165 291 291 291 F F F 18 360-379 162 162 162 288 288 288 F F F 19 380-399 162 162 162 288 288 288 F F F 20 400-419 165 165 165 291 291 291 F F F 21 420-439 162 162 162 288 288 288 F F F 22 440-459 156 156 156 277 277 277 F F F 23 460-479 165 165 165 291 291 291 F F F 24 480-499 162 162 162 288 288 288 F F F 25 500-519 162 162 162 288 288 288 F F F 26 520-539 165 165 165 291 291 291 F F F 27 540-559 160 160 160 283 283 283 F F F 28 560-579 165 165 165 291 291 291 F F F 29 580-599 165 165 165 291 291 291 F F F Max gen: 165, Max flush: 291 Begin loop 83 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.549757811Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:30.593439197Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.593839887Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.625158141Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.625710362Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.625771344Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.625791881Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.625819538Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870} {"msg":"Crucible 4338e869-90f0-42d1-b816-f981023ae66c has session id: fdc2d8b1-9841-45fa-8482-6973df5bb360","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.626460917Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"Upstairs opts: Upstairs UUID: 4338e869-90f0-42d1-b816-f981023ae66c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.626504891Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"Crucible stats registered with UUID: 4338e869-90f0-42d1-b816-f981023ae66c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.626525169Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:30.626541507Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.626804924Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.626842359Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"io task","client":"0","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.626877935Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"io task","client":"2","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.627031585Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"task":"crutest"} {"msg":"The guest has requested activation with gen:166","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.627057492Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870} {"msg":"4338e869-90f0-42d1-b816-f981023ae66c active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.627128003Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.62714794Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"client":"0","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.627169538Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"client":"1","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.627186685Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"client":"2","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.627203363Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"io task","client":"2","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.627220751Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"io task","client":"0","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.627241569Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"ds_connection connected from Ok(127.0.0.1:62579)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.627293562Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"io task","client":"2","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"ds_connection connected from Ok(127.0.0.1:59437)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.627331257Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"io task","client":"0","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"ds_connection connected from Ok(127.0.0.1:37015)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.627364323Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.628517368Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"client":"2","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.62857713Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"client":"1","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.628665559Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"client":"0","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.62906012Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.62913579Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"[0]R flush_numbers[0..12]: [279, 289, 291, 291, 284, 289, 289, 278, 289, 291, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629222369Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"downstairs","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"[0]R generation[0..12]: [158, 163, 165, 165, 161, 163, 163, 157, 163, 165, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629247336Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"downstairs","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629267813Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"downstairs","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"[1]R flush_numbers[0..12]: [279, 289, 291, 291, 284, 289, 289, 278, 289, 291, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629287071Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"downstairs","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"[1]R generation[0..12]: [158, 163, 165, 165, 161, 163, 163, 157, 163, 165, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629305818Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"downstairs","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629325516Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"downstairs","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"[2]R flush_numbers[0..12]: [279, 289, 291, 291, 284, 289, 289, 278, 289, 291, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629344934Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"downstairs","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"[2]R generation[0..12]: [158, 163, 165, 165, 161, 163, 163, 157, 163, 165, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629363651Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"downstairs","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629383199Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"downstairs","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"Max found gen is 166","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629399487Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"downstairs","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 166 >= found:166","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629422954Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"downstairs","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"Next flush: 292","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629442261Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"downstairs","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629458319Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"":"downstairs","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629475517Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629490865Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"client":"0","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629508013Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"client":"1","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629524751Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"client":"2","":"downstairs","session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629541969Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629558187Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"4338e869-90f0-42d1-b816-f981023ae66c is now active with session: fdc2d8b1-9841-45fa-8482-6973df5bb360","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629575755Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629590793Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870,"session_id":"fdc2d8b1-9841-45fa-8482-6973df5bb360"} {"msg":"The guest has finished waiting for activation with:166","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:18:30.629606621Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870} {"msg":"Activated sub_volume 4338e869-90f0-42d1-b816-f981023ae66c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:30.629621529Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 532 len 40960 data: 15 15 14 15 16 15 16 17 18 17 02/30 Read block 361 len 32768 03/30 Write block 284 len 36864 data: 12 11 11 12 10 19 21 18 16 04/30 Read block 142 len 36864 05/30 Write block 133 len 32768 data: 20 18 15 14 14 14 12 13 06/30 Flush 07/30 Write block 555 len 20480 data: 12 9 10 9 9 08/30 Read block 330 len 4096 09/30 Write block 244 len 12288 data: 19 20 18 10/30 Write block 497 len 4096 data: 7 11/30 Write block 349 len 32768 data: 11 13 10 10 14 12 12 11 12/30 Write block 146 len 4096 data: 7 13/30 Read block 356 len 28672 14/30 Flush 15/30 Write block 310 len 36864 data: 14 15 16 15 16 13 11 11 11 16/30 Write block 576 len 16384 data: 9 11 10 12 17/30 Read block 363 len 20480 18/30 Write block 497 len 32768 data: 8 9 12 10 12 12 12 10 19/30 Read block 585 len 8192 20/30 Flush 21/30 Write block 333 len 12288 data: 13 13 15 22/30 Read block 51 len 24576 23/30 Read block 52 len 28672 24/30 Read block 83 len 32768 25/30 Write block 583 len 12288 data: 9 8 13 26/30 Write block 33 len 24576 data: 14 16 19 20 19 23 27/30 Write block 542 len 40960 data: 21 17 17 15 12 13 14 12 10 12 28/30 Read block 273 len 8192 29/30 Write block 145 len 4096 data: 11 30/30 Write block 432 len 24576 data: 10 10 14 11 17 16 ---------------------------------------------------------------- Crucible gen:166 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 8 Done Done Done false 1002 Acked Write 9 Done Done Done false 1003 Acked Read 9 Done Done Done false 1004 Acked Write 8 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Read 1 Sent Done Done false 1008 Acked Write 3 Sent Done Done false 1009 Acked Write 1 Sent Done Done false 1010 Acked Write 8 Sent Done Done false 1011 Acked Write 1 Sent Done Done false 1012 Acked Read 7 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Write 4 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Read 6 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Read 8 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Write 1 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 1 1 28 Done 4 29 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:30.648946224Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1870} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 158 158 158 279 279 279 F F F 1 020-039 163 163 163 289 289 289 F T T <--- 2 040-059 165 165 165 291 291 291 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 161 161 161 284 284 284 F F F 5 100-119 163 163 163 289 289 289 F F F 6 120-139 163 166 166 289 292 292 F F F <--- 7 140-159 157 166 166 278 293 293 F T T <--- 8 160-179 163 163 163 289 289 289 F F F 9 180-199 165 165 165 291 291 291 F F F 10 200-219 158 158 158 279 279 279 F F F 11 220-239 156 156 156 277 277 277 F F F 12 240-259 159 166 166 281 293 293 F F F <--- 13 260-279 165 165 165 291 291 291 F F F 14 280-299 162 166 166 288 292 292 T F F <--- 15 300-319 159 166 166 281 294 294 F F F <--- 16 320-339 165 165 165 291 291 291 F T T <--- 17 340-359 165 166 166 291 293 293 F F F <--- 18 360-379 162 162 162 288 288 288 F F F 19 380-399 162 162 162 288 288 288 F F F 20 400-419 165 165 165 291 291 291 F F F 21 420-439 162 162 162 288 288 288 F T T <--- 22 440-459 156 156 156 277 277 277 F F F 23 460-479 165 165 165 291 291 291 F F F 24 480-499 162 166 166 288 294 294 F F F <--- 25 500-519 162 166 166 288 294 294 F F F <--- 26 520-539 165 166 166 291 292 292 T F F <--- 27 540-559 160 166 166 283 293 293 T T T <--- 28 560-579 165 166 166 291 294 294 F F F <--- 29 580-599 165 165 165 291 291 291 F T T <--- Max gen: 166, Max flush: 294 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 83 Verifying data now No change after verify Loop: 83 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 158 158 158 279 279 279 F F F 1 020-039 167 167 167 295 295 295 F F F 2 040-059 165 165 165 291 291 291 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 161 161 161 284 284 284 F F F 5 100-119 163 163 163 289 289 289 F F F 6 120-139 166 166 166 292 292 292 F F F 7 140-159 167 167 167 295 295 295 F F F 8 160-179 163 163 163 289 289 289 F F F 9 180-199 165 165 165 291 291 291 F F F 10 200-219 158 158 158 279 279 279 F F F 11 220-239 156 156 156 277 277 277 F F F 12 240-259 166 166 166 293 293 293 F F F 13 260-279 165 165 165 291 291 291 F F F 14 280-299 166 166 166 292 292 292 F F F 15 300-319 166 166 166 294 294 294 F F F 16 320-339 167 167 167 295 295 295 F F F 17 340-359 166 166 166 293 293 293 F F F 18 360-379 162 162 162 288 288 288 F F F 19 380-399 162 162 162 288 288 288 F F F 20 400-419 165 165 165 291 291 291 F F F 21 420-439 167 167 167 295 295 295 F F F 22 440-459 156 156 156 277 277 277 F F F 23 460-479 165 165 165 291 291 291 F F F 24 480-499 166 166 166 294 294 294 F F F 25 500-519 166 166 166 294 294 294 F F F 26 520-539 166 166 166 292 292 292 F F F 27 540-559 167 167 167 295 295 295 F F F 28 560-579 166 166 166 294 294 294 F F F 29 580-599 167 167 167 295 295 295 F F F Max gen: 167, Max flush: 295 Begin loop 84 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.105347131Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.150636995Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.151251107Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.1513118Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.151343166Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.151371682Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878} {"msg":"Crucible 0cd692ba-4c31-4f0b-b400-e09514dabc26 has session id: 7cda820c-7937-45c8-9628-1bed397a2d22","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.151928872Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"Upstairs opts: Upstairs UUID: 0cd692ba-4c31-4f0b-b400-e09514dabc26, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.151978806Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"Crucible stats registered with UUID: 0cd692ba-4c31-4f0b-b400-e09514dabc26","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.152000273Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:35.152016951Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.152204317Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"0","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.152243713Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.15226402Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.152444017Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"task":"crutest"} {"msg":"The guest has requested activation with gen:168","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.152467474Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878} {"msg":"0cd692ba-4c31-4f0b-b400-e09514dabc26 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.152534486Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.152553923Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"client":"0","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.152575231Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"client":"1","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.152604567Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"client":"2","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.152675198Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.152710184Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"0","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.15273917Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"ds_connection connected from Ok(127.0.0.1:62573)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.152767287Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"ds_connection connected from Ok(127.0.0.1:49185)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.152798343Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"0","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:35.152838778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:35.152905739Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:35.153019095Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.15305832Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.153151718Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.15408904Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"client":"2","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.154122176Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"client":"0","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.15472618Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:35.154773454Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.154223382Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"ds_connection connected from Ok(127.0.0.1:43411)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.154332928Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.156157458Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"client":"1","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"[0]R flush_numbers[0..12]: [279, 295, 291, 291, 284, 289, 292, 295, 289, 291, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.156752723Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"downstairs","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"[0]R generation[0..12]: [158, 167, 165, 165, 161, 163, 166, 167, 163, 165, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.15677771Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"downstairs","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.156798488Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"downstairs","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"[1]R flush_numbers[0..12]: [279, 295, 291, 291, 284, 289, 292, 295, 289, 291, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.156818175Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"downstairs","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"[1]R generation[0..12]: [158, 167, 165, 165, 161, 163, 166, 167, 163, 165, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.156837033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"downstairs","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.15685669Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"downstairs","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"[2]R flush_numbers[0..12]: [279, 295, 291, 291, 284, 289, 292, 295, 289, 291, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.156875898Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"downstairs","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"[2]R generation[0..12]: [158, 167, 165, 165, 161, 163, 166, 167, 163, 165, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.156895015Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"downstairs","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.156914483Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"downstairs","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"Max found gen is 168","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.15693765Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"downstairs","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"Generation requested: 168 >= found:168","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO ,"time":"2025-11-06T20:18:45.156968026Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"downstairs","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"Next flush: 296","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.157004202Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"downstairs","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.157024469Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"":"downstairs","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.157051176Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.157079512Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"client":"0","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.15709987Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"client":"1","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.157116878Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"client":"2","":"downstairs","session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.157134085Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.157150103Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"0cd692ba-4c31-4f0b-b400-e09514dabc26 is now active with session: 7cda820c-7937-45c8-9628-1bed397a2d22","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:18:45.157167921Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.157187209Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878,"session_id":"7cda820c-7937-45c8-9628-1bed397a2d22"} {"msg":"The guest has finished waiting for activation with:168","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.157203487Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878} {"msg":"Activated sub_volume 0cd692ba-4c31-4f0b-b400-e09514dabc26","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:45.157217665Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 293 len 40960 data: 18 18 16 14 12 13 12 9 10 11 02/30 Write block 532 len 24576 data: 16 16 15 16 17 16 03/30 Write block 98 len 36864 data: 13 14 10 10 11 9 10 11 11 04/30 Write block 498 len 24576 data: 10 13 11 13 13 13 05/30 Write block 279 len 4096 data: 13 06/30 Write block 573 len 8192 data: 11 9 07/30 Read block 12 len 32768 08/30 Read block 5 len 4096 09/30 Read block 137 len 4096 10/30 Write block 576 len 28672 data: 10 12 11 13 9 9 9 11/30 Write block 298 len 8192 data: 14 13 12/30 Read block 579 len 4096 13/30 Read block 133 len 28672 14/30 Read block 417 len 24576 15/30 Write block 447 len 4096 data: 10 16/30 Read block 328 len 40960 17/30 Write block 118 len 20480 data: 17 18 19 14 14 18/30 Flush 19/30 Write block 443 len 8192 data: 12 14 20/30 Read block 304 len 40960 21/30 Read block 20 len 36864 22/30 Write block 311 len 32768 data: 16 17 16 17 14 12 12 12 23/30 Read block 494 len 20480 24/30 Write block 25 len 8192 data: 18 17 25/30 Read block 117 len 16384 26/30 Write block 327 len 36864 data: 12 12 10 10 10 10 14 14 16 27/30 Write block 496 len 8192 data: 5 9 28/30 Write block 553 len 24576 data: 14 13 13 10 11 10 29/30 Read block 586 len 12288 30/30 Read block 551 len 4096 ---------------------------------------------------------------- Crucible gen:168 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 6 Done Sent Done false 1002 Acked Write 9 Done Sent Done false 1003 Acked Write 6 Done Sent Done false 1004 Acked Write 1 Done Sent Done false 1005 Acked Write 2 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Read 1 Done Sent Done false 1008 Acked Read 1 Done Sent Done false 1009 Acked Write 7 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Read 1 Done Sent Done false 1012 Acked Read 7 Done Sent Done false 1013 Acked Read 6 Done Sent Done false 1014 Acked Write 1 Done Sent Done false 1015 Acked Read 10 Done Sent Done false 1016 Acked Write 5 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 2 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Write 8 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Write 2 Done Sent Done false 1024 Acked Read 4 Done Sent Done false 1025 Acked Write 9 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Write 6 Done Sent Done false 1028 Acked Read 3 Done Sent Done false 1029 Acked Read 1 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) None Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:45.175130438Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1878} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 158 158 158 279 279 279 F F F 1 020-039 167 167 167 295 295 295 T F T <--- 2 040-059 165 165 165 291 291 291 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 168 161 168 296 284 296 F F F <--- 5 100-119 168 163 168 296 289 296 F F F <--- 6 120-139 168 166 168 296 292 296 F F F <--- 7 140-159 167 167 167 295 295 295 F F F 8 160-179 163 163 163 289 289 289 F F F 9 180-199 165 165 165 291 291 291 F F F 10 200-219 158 158 158 279 279 279 F F F 11 220-239 156 156 156 277 277 277 F F F 12 240-259 166 166 166 293 293 293 F F F 13 260-279 168 165 168 296 291 296 F F F <--- 14 280-299 168 166 168 296 292 296 F T F <--- 15 300-319 168 166 168 296 294 296 T T T <--- 16 320-339 167 167 167 295 295 295 T F T <--- 17 340-359 166 166 166 293 293 293 F F F 18 360-379 162 162 162 288 288 288 F F F 19 380-399 162 162 162 288 288 288 F F F 20 400-419 165 165 165 291 291 291 F F F 21 420-439 167 167 167 295 295 295 F F F 22 440-459 168 156 168 296 277 296 T F T <--- 23 460-479 165 165 165 291 291 291 F F F 24 480-499 168 166 168 296 294 296 T F T <--- 25 500-519 168 166 168 296 294 296 F F F <--- 26 520-539 168 166 168 296 292 296 F F F <--- 27 540-559 167 167 167 295 295 295 T F T <--- 28 560-579 168 166 168 296 294 296 F F F <--- 29 580-599 168 167 168 296 295 296 F F F <--- Max gen: 168, Max flush: 296 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 84 Verifying data now No change after verify Loop: 84 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 158 158 158 279 279 279 F F F 1 020-039 169 169 169 297 297 297 F F F 2 040-059 165 165 165 291 291 291 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 168 168 168 296 296 296 F F F 5 100-119 168 168 168 296 296 296 F F F 6 120-139 168 168 168 296 296 296 F F F 7 140-159 167 167 167 295 295 295 F F F 8 160-179 163 163 163 289 289 289 F F F 9 180-199 165 165 165 291 291 291 F F F 10 200-219 158 158 158 279 279 279 F F F 11 220-239 156 156 156 277 277 277 F F F 12 240-259 166 166 166 293 293 293 F F F 13 260-279 168 168 168 296 296 296 F F F 14 280-299 168 168 168 296 296 296 F F F 15 300-319 169 169 169 297 297 297 F F F 16 320-339 169 169 169 297 297 297 F F F 17 340-359 166 166 166 293 293 293 F F F 18 360-379 162 162 162 288 288 288 F F F 19 380-399 162 162 162 288 288 288 F F F 20 400-419 165 165 165 291 291 291 F F F 21 420-439 167 167 167 295 295 295 F F F 22 440-459 169 169 169 297 297 297 F F F 23 460-479 165 165 165 291 291 291 F F F 24 480-499 169 169 169 297 297 297 F F F 25 500-519 168 168 168 296 296 296 F F F 26 520-539 168 168 168 296 296 296 F F F 27 540-559 169 169 169 297 297 297 F F F 28 560-579 168 168 168 296 296 296 F F F 29 580-599 168 168 168 296 296 296 F F F Max gen: 169, Max flush: 297 Begin loop 85 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.6149742Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.660269003Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.660887356Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.66093495Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.660954937Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.660977284Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886} {"msg":"Crucible 6e6e1fc2-9d34-4949-a341-b41c89adab05 has session id: fb200d6d-356f-4bb7-b564-dce9d2963432","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.661475051Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"Upstairs opts: Upstairs UUID: 6e6e1fc2-9d34-4949-a341-b41c89adab05, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.661526995Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"Crucible stats registered with UUID: 6e6e1fc2-9d34-4949-a341-b41c89adab05","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.661548252Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:49.66156516Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.661782803Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"io task","client":"0","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.661826497Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"io task","client":"1","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.661846175Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.662081385Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"task":"crutest"} {"msg":"The guest has requested activation with gen:170","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.662108292Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886} {"msg":"6e6e1fc2-9d34-4949-a341-b41c89adab05 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.662178943Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.66219852Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"client":"0","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.662222057Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"client":"1","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.662252254Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"client":"2","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.662291299Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.662325295Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"io task","client":"0","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.66235891Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"io task","client":"1","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"ds_connection connected from Ok(127.0.0.1:58633)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.662393056Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"ds_connection connected from Ok(127.0.0.1:49105)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.662428682Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"io task","client":"0","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"ds_connection connected from Ok(127.0.0.1:45689)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.662461577Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"io task","client":"1","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.663662016Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"client":"1","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.66370941Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"client":"0","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.664332082Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.664372597Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.664779415Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"client":"2","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"[0]R flush_numbers[0..12]: [279, 297, 291, 291, 296, 296, 296, 295, 289, 291, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665276563Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"downstairs","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"[0]R generation[0..12]: [158, 169, 165, 165, 168, 168, 168, 167, 163, 165, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665302149Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"downstairs","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665323027Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"downstairs","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"[1]R flush_numbers[0..12]: [279, 297, 291, 291, 296, 296, 296, 295, 289, 291, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665346674Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"downstairs","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"[1]R generation[0..12]: [158, 169, 165, 165, 168, 168, 168, 167, 163, 165, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665366411Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"downstairs","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665386809Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"downstairs","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"[2]R flush_numbers[0..12]: [279, 297, 291, 291, 296, 296, 296, 295, 289, 291, 279, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665406506Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"downstairs","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"[2]R generation[0..12]: [158, 169, 165, 165, 168, 168, 168, 167, 163, 165, 158, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665425554Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"downstairs","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665446151Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"downstairs","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"Max found gen is 170","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665462349Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"downstairs","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"Generation requested: 170 >= found:170","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665479177Z","hostname":"Wait for a query_work_queue command to finish before sending IO w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"downstairs","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"Next flush: 298","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665502624Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"downstairs","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665518642Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"":"downstairs","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.66553652Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665552028Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"client":"0","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665569576Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"client":"1","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665586453Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"client":"2","":"downstairs","session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665603251Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665621629Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"6e6e1fc2-9d34-4949-a341-b41c89adab05 is now active with session: fb200d6d-356f-4bb7-b564-dce9d2963432","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665640527Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665656095Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886,"session_id":"fb200d6d-356f-4bb7-b564-dce9d2963432"} {"msg":"The guest has finished waiting for activation with:170","vDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.665676722Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886} {"msg":"Activated sub_volume 6e6e1fc2-9d34-4949-a341-b41c89adab05","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:49.66569197Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 514 len 24576 data: 13 14 13 14 13 11 02/30 Write block 563 len 8192 data: 9 12 03/30 Read block 94 len 16384 04/30 Write block 545 len 32768 data: 16 13 14 15 13 11 13 13 05/30 Flush 06/30 Write block 198 len 16384 data: 15 14 15 17 07/30 Read block 395 len 16384 08/30 Read block 485 len 28672 09/30 Flush 10/30 Write block 95 len 12288 data: 16 14 14 11/30 Write block 443 len 24576 data: 13 15 15 10 11 14 12/30 Flush 13/30 Write block 211 len 4096 data: 9 14/30 Write block 320 len 4096 data: 11 15/30 Read block 17 len 32768 16/30 Write block 319 len 16384 data: 12 12 13 8 17/30 Flush 18/30 Write block 243 len 12288 data: 19 20 21 19/30 Write block 418 len 8192 data: 19 17 20/30 Read block 144 len 24576 21/30 Read block 384 len 4096 22/30 Flush 23/30 Write block 406 len 12288 data: 12 12 13 24/30 Write block 347 len 32768 data: 10 11 12 14 11 11 15 13 25/30 Write block 348 len 40960 data: 12 13 15 12 12 16 14 13 12 11 26/30 Read block 364 len 8192 27/30 Write block 368 len 4096 data: 11 28/30 Write block 2 len 20480 data: 2 3 4 5 7 29/30 Read block 375 len 8192 30/30 Write block 481 len 8192 data: 13 12 ---------------------------------------------------------------- Crucible gen:170 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Sent false 1001 Acked Write 2 Done Done Sent false 1002 Acked Read 4 Done Done Sent false 1003 Acked Write 8 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 4 Done Done Sent false 1006 Acked Read 4 Done Done Sent false 1007 Acked Read 7 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Write 6 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 1 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Read 8 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 3 Done Done Sent false 1018 Acked Write 2 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Read 1 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Write 8 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Read 2 Done Done Sent false 1026 Acked Write 1 Done Done Sent false 1027 Acked Write 5 Done Done Sent false 1028 Acked Read 2 Done Done Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 30 32 Done 29 29 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1021)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:49.683438304Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1886} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 158 158 158 279 279 279 T T F <--- 1 020-039 169 169 169 297 297 297 F F F 2 040-059 165 165 165 291 291 291 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 170 170 168 300 300 296 F F F <--- 5 100-119 168 168 168 296 296 296 F F F 6 120-139 168 168 168 296 296 296 F F F 7 140-159 167 167 167 295 295 295 F F F 8 160-179 163 163 163 289 289 289 F F F 9 180-199 170 170 165 299 299 291 F F F <--- 10 200-219 170 170 158 301 301 279 F F F <--- 11 220-239 156 156 156 277 277 277 F F F 12 240-259 170 170 166 302 302 293 F F F <--- 13 260-279 168 168 168 296 296 296 F F F 14 280-299 168 168 168 296 296 296 F F F 15 300-319 170 170 169 301 301 297 F F F <--- 16 320-339 170 170 169 301 301 297 F F F <--- 17 340-359 166 166 166 293 293 293 T T F <--- 18 360-379 162 162 162 288 288 288 T T F <--- 19 380-399 162 162 162 288 288 288 F F F 20 400-419 170 170 165 302 302 291 T T F <--- 21 420-439 167 167 167 295 295 295 F F F 22 440-459 170 170 169 300 300 297 F F F <--- 23 460-479 165 165 165 291 291 291 F F F 24 480-499 169 169 169 297 297 297 T T F <--- 25 500-519 170 170 168 298 298 296 F F F <--- 26 520-539 168 168 168 296 296 296 F F F 27 540-559 170 170 169 298 298 297 F F F <--- 28 560-579 170 170 168 298 298 296 F F F <--- 29 580-599 168 168 168 296 296 296 F F F Max gen: 170, Max flush: 302 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 85 Verifying data now No change after verify Loop: 85 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 171 171 171 303 303 303 F F F 1 020-039 169 169 169 297 297 297 F F F 2 040-059 165 165 165 291 291 291 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 170 170 170 300 300 300 F F F 5 100-119 168 168 168 296 296 296 F F F 6 120-139 168 168 168 296 296 296 F F F 7 140-159 167 167 167 295 295 295 F F F 8 160-179 163 163 163 289 289 289 F F F 9 180-199 170 170 170 299 299 299 F F F 10 200-219 170 170 170 301 301 301 F F F 11 220-239 156 156 156 277 277 277 F F F 12 240-259 170 170 170 302 302 302 F F F 13 260-279 168 168 168 296 296 296 F F F 14 280-299 168 168 168 296 296 296 F F F 15 300-319 170 170 170 301 301 301 F F F 16 320-339 170 170 170 301 301 301 F F F 17 340-359 171 171 171 303 303 303 F F F 18 360-379 171 171 171 303 303 303 F F F 19 380-399 162 162 162 288 288 288 F F F 20 400-419 171 171 171 303 303 303 F F F 21 420-439 167 167 167 295 295 295 F F F 22 440-459 170 170 170 300 300 300 F F F 23 460-479 165 165 165 291 291 291 F F F 24 480-499 171 171 171 303 303 303 F F F 25 500-519 170 170 170 298 298 298 F F F 26 520-539 168 168 168 296 296 296 F F F 27 540-559 170 170 170 298 298 298 F F F 28 560-579 170 170 170 298 298 298 F F F 29 580-599 168 168 168 296 296 296 F F F Max gen: 171, Max flush: 303 Begin loop 86 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.09667504Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.14190971Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.142564978Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.14262454Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.142644778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.142672464Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894} {"msg":"Crucible ad7b382d-b754-4042-bb04-474faf63ca7f has session id: daf71f15-4185-4a46-892c-a41bafafa9a0","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.14310775Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"Upstairs opts: Upstairs UUID: ad7b382d-b754-4042-bb04-474faf63ca7f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.143254431Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"Crucible stats registered with UUID: ad7b382d-b754-4042-bb04-474faf63ca7f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.143295066Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:54.143324592Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.143519588Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"2","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.143568301Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"0","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.143600887Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.143784874Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"task":"crutest"} {"msg":"The guest has requested activation with gen:172","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.143813681Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894} {"msg":"ad7b382d-b754-4042-bb04-474faf63ca7f active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.143874823Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.14389532Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"client":"0","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.143913368Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"client":"1","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.143930486Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"client":"2","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.143947314Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"2","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.143964611Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"0","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.14398152Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"ds_connection connected from Ok(127.0.0.1:61795)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.144075928Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"0","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"ds_connection connected from Ok(127.0.0.1:47369)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.144110303Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"2","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:54.144284671Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:54.144353813Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:18:54.144410275Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.144448171Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"sleeping for 10s before connecting","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.144485696Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.145233162Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"client":"0","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.145343528Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"client":"2","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.145765645Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:18:54.145878681Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.145097268Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"ds_connection connected from Ok(127.0.0.1:54893)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.145197595Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.146701466Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"client":"1","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"[0]R flush_numbers[0..12]: [303, 297, 291, 291, 300, 296, 296, 295, 289, 299, 301, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147425205Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"downstairs","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"[0]R generation[0..12]: [171, 169, 165, 165, 170, 168, 168, 167, 163, 170, 170, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.14745782Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"downstairs","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147479268Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"downstairs","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"[1]R flush_numbers[0..12]: [303, 297, 291, 291, 300, 296, 296, 295, 289, 299, 301, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147502485Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"downstairs","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"[1]R generation[0..12]: [171, 169, 165, 165, 170, 168, 168, 167, 163, 170, 170, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147522102Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"downstairs","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.14754196Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"downstairs","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"[2]R flush_numbers[0..12]: [303, 297, 291, 291, 300, 296, 296, 295, 289, 299, 301, 277]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147561097Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"downstairs","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"[2]R generation[0..12]: [171, 169, 165, 165, 170, 168, 168, 167, 163, 170, 170, 156]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147580005Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"downstairs","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147599503Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"downstairs","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"Max found gen is 172","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147615901Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"downstairs","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"Generation requested: 172 >= found:172","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147632998Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"downstairs","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"Next flush: 304","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147649106Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"downstairs","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO 2025-11-06T20:19:04.147665334Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"":"downstairs","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147691111Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147706589Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"client":"0","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147723747Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"client":"1","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147741015Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"client":"2","":"downstairs","session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147758643Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.14777687Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"ad7b382d-b754-4042-bb04-474faf63ca7f is now active with session: daf71f15-4185-4a46-892c-a41bafafa9a0","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147794888Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147809876Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894,"session_id":"daf71f15-4185-4a46-892c-a41bafafa9a0"} {"msg":"The guest has finished waiting for activation with:172","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147826064Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894} {"msg":"Activated sub_volume ad7b382d-b754-4042-bb04-474faf63ca7f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:04.147839822Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 337 len 12288 data: 16 15 13 02/30 Flush 03/30 Write block 352 len 16384 data: 13 17 15 14 04/30 Write block 584 len 32768 data: 9 14 13 14 18 21 21 18 05/30 Read block 395 len 32768 06/30 Read block 36 len 24576 07/30 Read block 372 len 32768 08/30 Read block 409 len 20480 09/30 Read block 7 len 8192 10/30 Flush 11/30 Flush 12/30 Write block 455 len 4096 data: 20 13/30 Write block 348 len 16384 data: 13 14 16 13 14/30 Read block 360 len 12288 15/30 Flush 16/30 Write block 232 len 16384 data: 13 14 11 13 17/30 Write block 418 len 4096 data: 20 18/30 Write block 377 len 20480 data: 9 10 11 14 13 19/30 Write block 266 len 32768 data: 15 17 14 15 14 16 16 14 20/30 Flush 21/30 Write block 556 len 16384 data: 11 12 11 10 22/30 Flush 23/30 Write block 565 len 8192 data: 10 9 24/30 Read block 495 len 40960 25/30 Write block 249 len 4096 data: 14 26/30 Read block 128 len 28672 27/30 Read block 121 len 40960 28/30 Read block 49 len 40960 29/30 Read block 460 len 24576 30/30 Read block 531 len 40960 ---------------------------------------------------------------- Crucible gen:172 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 4 Done Done Done false 1003 Acked Write 8 Done Done Done false 1004 Acked Read 8 Done Done Done false 1005 Acked Read 6 Done Done Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Flush 0 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Write 4 Done Sent Done false 1013 Acked Read 3 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 4 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Write 5 Done Sent Done false 1018 Acked Write 8 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 4 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 2 Done Sent Done false 1023 Acked Read 10 Done Sent Done false 1024 Acked Write 1 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Read 10 Done Sent Done false 1027 Acked Read 10 Done Sent Done false 1028 Acked Read 6 Done Sent Done false 1029 Acked Read 10 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 24 0 24 Done 28 4 28 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1001)) Some(JobId(1021)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:04.166531197Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1894} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 171 171 171 303 303 303 F F F 1 020-039 169 169 169 297 297 297 F F F 2 040-059 165 165 165 291 291 291 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 170 170 170 300 300 300 F F F 5 100-119 168 168 168 296 296 296 F F F 6 120-139 168 168 168 296 296 296 F F F 7 140-159 167 167 167 295 295 295 F F F 8 160-179 163 163 163 289 289 289 F F F 9 180-199 170 170 170 299 299 299 F F F 10 200-219 170 170 170 301 301 301 F F F 11 220-239 172 156 172 308 277 308 F F F <--- 12 240-259 170 170 170 302 302 302 T F T <--- 13 260-279 172 168 172 308 296 308 F F F <--- 14 280-299 168 168 168 296 296 296 F F F 15 300-319 170 170 170 301 301 301 F F F 16 320-339 172 172 172 304 304 304 F F F 17 340-359 172 171 172 307 303 307 F T F <--- 18 360-379 172 171 172 308 303 308 F F F <--- 19 380-399 172 162 172 308 288 308 F F F <--- 20 400-419 172 171 172 308 303 308 F F F <--- 21 420-439 167 167 167 295 295 295 F F F 22 440-459 172 170 172 307 300 307 F F F <--- 23 460-479 165 165 165 291 291 291 F F F 24 480-499 171 171 171 303 303 303 F F F 25 500-519 170 170 170 298 298 298 F F F 26 520-539 168 168 168 296 296 296 F F F 27 540-559 172 170 172 309 298 309 F F F <--- 28 560-579 170 170 170 298 298 298 T F T <--- 29 580-599 172 168 172 305 296 305 F T F <--- Max gen: 172, Max flush: 309 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 86 Verifying data now No change after verify Loop: 86 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 171 171 171 303 303 303 F F F 1 020-039 169 169 169 297 297 297 F F F 2 040-059 165 165 165 291 291 291 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 170 170 170 300 300 300 F F F 5 100-119 168 168 168 296 296 296 F F F 6 120-139 168 168 168 296 296 296 F F F 7 140-159 167 167 167 295 295 295 F F F 8 160-179 163 163 163 289 289 289 F F F 9 180-199 170 170 170 299 299 299 F F F 10 200-219 170 170 170 301 301 301 F F F 11 220-239 172 172 172 308 308 308 F F F 12 240-259 173 173 173 310 310 310 F F F 13 260-279 172 172 172 308 308 308 F F F 14 280-299 168 168 168 296 296 296 F F F 15 300-319 170 170 170 301 301 301 F F F 16 320-339 172 172 172 304 304 304 F F F 17 340-359 172 172 172 307 307 307 F F F 18 360-379 172 172 172 308 308 308 F F F 19 380-399 172 172 172 308 308 308 F F F 20 400-419 172 172 172 308 308 308 F F F 21 420-439 167 167 167 295 295 295 F F F 22 440-459 172 172 172 307 307 307 F F F 23 460-479 165 165 165 291 291 291 F F F 24 480-499 171 171 171 303 303 303 F F F 25 500-519 170 170 170 298 298 298 F F F 26 520-539 168 168 168 296 296 296 F F F 27 540-559 172 172 172 309 309 309 F F F 28 560-579 173 173 173 310 310 310 F F F 29 580-599 172 172 172 305 305 305 F F F Max gen: 173, Max flush: 310 Begin loop 87 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.566623609Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.61423412Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.614909815Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.614959309Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.614976077Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.615002553Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902} {"msg":"Crucible ddac3275-62fa-4767-9575-855b01270979 has session id: bf9429c7-f3b1-4948-ba10-6e4bb18942bb","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.61550336Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"Upstairs opts: Upstairs UUID: ddac3275-62fa-4767-9575-855b01270979, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.615551484Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"Crucible stats registered with UUID: ddac3275-62fa-4767-9575-855b01270979","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.615573261Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:08.615598718Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.615814321Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"io task","client":"0","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.615866425Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.61590016Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"io task","client":"2","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.616086607Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"task":"crutest"} {"msg":"The guest has requested activation with gen:174","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.616119913Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902} {"msg":"ddac3275-62fa-4767-9575-855b01270979 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.616183805Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.616205192Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"client":"0","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.61622319Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"client":"1","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.616240277Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"client":"2","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.616258325Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"io task","client":"2","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.616285132Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"io task","client":"0","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.616314908Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:49269)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.616345644Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"io task","client":"2","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:37490)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.61637674Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"io task","client":"0","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:33672)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.616570686Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.617617534Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"client":"0","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.617661029Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"client":"2","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.618238796Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.618269222Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.618640445Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"client":"1","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"[0]R flush_numbers[0..12]: [303, 297, 291, 291, 300, 296, 296, 295, 289, 299, 301, 308]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619166849Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"downstairs","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"[0]R generation[0..12]: [171, 169, 165, 165, 170, 168, 168, 167, 163, 170, 170, 172]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619192176Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"downstairs","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619212803Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"downstairs","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"[1]R flush_numbers[0..12]: [303, 297, 291, 291, 300, 296, 296, 295, 289, 299, 301, 308]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619231931Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"downstairs","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"[1]R generation[0..12]: [171, 169, 165, 165, 170, 168, 168, 167, 163, 170, 170, 172]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619250548Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"downstairs","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619270106Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"downstairs","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"[2]R flush_numbers[0..12]: [303, 297, 291, 291, 300, 296, 296, 295, 289, 299, 301, 308]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619289104Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"downstairs","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"[2]R generation[0..12]: [171, 169, 165, 165, 170, 168, 168, 167, 163, 170, 170, 172]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619307811Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"downstairs","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619327359Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"downstairs","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"Max found gen is 174","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619343577Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"downstairs","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"Wait for a query_work_queue command to finish before sending IO Generation requested: 174 >= found:174","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619369073Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"downstairs","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"Next flush: 311","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619385821Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"downstairs","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619402089Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"":"downstairs","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619419037Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619434355Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"client":"0","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619451163Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"client":"1","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619467791Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"client":"2","":"downstairs","session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619484739Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619500657Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"ddac3275-62fa-4767-9575-855b01270979 is now active with session: bf9429c7-f3b1-4948-ba10-6e4bb18942bb","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619518555Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619533663Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902,"session_id":"bf9429c7-f3b1-4948-ba10-6e4bb18942bb"} {"msg":"The guest has finished waiting for activation with:174","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"time":"2025-11-06T20:19:08.61954978Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902} {"msg":"Activated sub_volume ddac3275-62fa-4767-9575-855b01270979","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:08.619568618Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 125 len 20480 data: 16 15 15 16 19 02/30 Write block 438 len 20480 data: 14 16 13 14 13 03/30 Read block 571 len 24576 04/30 Read block 101 len 36864 05/30 Flush 06/30 Write block 143 len 4096 data: 13 07/30 Write block 153 len 20480 data: 10 12 13 12 13 08/30 Read block 89 len 32768 09/30 Read block 147 len 8192 10/30 Write block 39 len 40960 data: 20 22 23 20 17 17 18 18 18 15 11/30 Read block 222 len 36864 12/30 Read block 407 len 20480 13/30 Write block 588 len 12288 data: 19 22 22 14/30 Write block 527 len 24576 data: 10 11 10 11 14 17 15/30 Write block 288 len 24576 data: 11 20 22 19 17 19 16/30 Write block 203 len 24576 data: 16 16 18 14 13 11 17/30 Write block 478 len 4096 data: 15 18/30 Read block 534 len 12288 19/30 Write block 172 len 8192 data: 8 7 20/30 Write block 230 len 4096 data: 11 21/30 Write block 454 len 36864 data: 21 21 19 17 14 14 9 10 11 22/30 Write block 211 len 12288 data: 10 9 9 23/30 Read block 108 len 20480 24/30 Write block 543 len 16384 data: 18 18 17 14 25/30 Write block 453 len 16384 data: 20 22 22 20 26/30 Write block 429 len 28672 data: 14 15 12 11 11 15 12 27/30 Read block 587 len 8192 28/30 Write block 93 len 4096 data: 14 29/30 Read block 205 len 24576 30/30 Read block 134 len 40960 ---------------------------------------------------------------- Crucible gen:174 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 5 Done Done Done false 1002 Acked Read 6 Done Done Done false 1003 Acked Read 9 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 1 Done Sent Done false 1006 Acked Write 5 Done Sent Done false 1007 Acked Read 8 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Write 10 Done Sent Done false 1010 Acked Read 9 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Write 3 Done Sent Done false 1013 Acked Write 6 Done Sent Done false 1014 Acked Write 6 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Read 3 Done Sent Done false 1018 Acked Write 2 Done Sent Done false 1019 Acked Write 1 Done Sent Done false 1020 Acked Write 9 Done Sent Done false 1021 Acked Write 3 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Write 4 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Read 6 Done Sent Done false 1029 Acked Read 10 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 0 28 Done 29 3 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1004)) None Some(JobId(1004)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:08.636167537Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1902} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 171 171 171 303 303 303 F F F 1 020-039 169 169 169 297 297 297 T F T <--- 2 040-059 165 165 165 291 291 291 T F T <--- 3 060-079 165 165 165 291 291 291 F F F 4 080-099 170 170 170 300 300 300 T F T <--- 5 100-119 168 168 168 296 296 296 F F F 6 120-139 174 168 174 311 296 311 F T F <--- 7 140-159 167 167 167 295 295 295 T F T <--- 8 160-179 163 163 163 289 289 289 T F T <--- 9 180-199 170 170 170 299 299 299 F F F 10 200-219 170 170 170 301 301 301 T F T <--- 11 220-239 172 172 172 308 308 308 T F T <--- 12 240-259 173 173 173 310 310 310 F F F 13 260-279 172 172 172 308 308 308 F F F 14 280-299 168 168 168 296 296 296 T F T <--- 15 300-319 170 170 170 301 301 301 F F F 16 320-339 172 172 172 304 304 304 F F F 17 340-359 172 172 172 307 307 307 F F F 18 360-379 172 172 172 308 308 308 F F F 19 380-399 172 172 172 308 308 308 F F F 20 400-419 172 172 172 308 308 308 F F F 21 420-439 174 167 174 311 295 311 T T T <--- 22 440-459 174 172 174 311 307 311 T T T <--- 23 460-479 165 165 165 291 291 291 T F T <--- 24 480-499 171 171 171 303 303 303 F F F 25 500-519 170 170 170 298 298 298 F F F 26 520-539 168 168 168 296 296 296 T F T <--- 27 540-559 172 172 172 309 309 309 T F T <--- 28 560-579 173 173 173 310 310 310 F F F 29 580-599 172 172 172 305 305 305 T F T <--- Max gen: 174, Max flush: 311 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 87 Verifying data now No change after verify Loop: 87 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 171 171 171 303 303 303 F F F 1 020-039 175 175 175 312 312 312 F F F 2 040-059 175 175 175 312 312 312 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 175 175 175 312 312 312 F F F 5 100-119 168 168 168 296 296 296 F F F 6 120-139 174 174 174 311 311 311 F F F 7 140-159 175 175 175 312 312 312 F F F 8 160-179 175 175 175 312 312 312 F F F 9 180-199 170 170 170 299 299 299 F F F 10 200-219 175 175 175 312 312 312 F F F 11 220-239 175 175 175 312 312 312 F F F 12 240-259 173 173 173 310 310 310 F F F 13 260-279 172 172 172 308 308 308 F F F 14 280-299 175 175 175 312 312 312 F F F 15 300-319 170 170 170 301 301 301 F F F 16 320-339 172 172 172 304 304 304 F F F 17 340-359 172 172 172 307 307 307 F F F 18 360-379 172 172 172 308 308 308 F F F 19 380-399 172 172 172 308 308 308 F F F 20 400-419 172 172 172 308 308 308 F F F 21 420-439 175 175 175 312 312 312 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 175 175 175 312 312 312 F F F 24 480-499 171 171 171 303 303 303 F F F 25 500-519 170 170 170 298 298 298 F F F 26 520-539 175 175 175 312 312 312 F F F 27 540-559 175 175 175 312 312 312 F F F 28 560-579 173 173 173 310 310 310 F F F 29 580-599 175 175 175 312 312 312 F F F Max gen: 175, Max flush: 312 Begin loop 88 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.05926504Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.104232834Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.104841018Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.104895001Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.104920157Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.104948684Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910} {"msg":"Crucible c298541d-e00b-49cc-8c0d-ec55a7f5e5df has session id: f554a9d6-a5f0-48f2-8ac3-253aa917e685","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.105523641Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"Upstairs opts: Upstairs UUID: c298541d-e00b-49cc-8c0d-ec55a7f5e5df, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.105567526Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"Crucible stats registered with UUID: c298541d-e00b-49cc-8c0d-ec55a7f5e5df","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.105587804Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:13.105604411Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.105841561Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"io task","client":"0","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.105877627Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"io task","client":"2","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.105896254Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.106073472Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"task":"crutest"} {"msg":"The guest has requested activation with gen:176","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.106097689Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910} {"msg":"c298541d-e00b-49cc-8c0d-ec55a7f5e5df active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.106184078Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.106204656Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"client":"0","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.106224823Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"client":"1","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.106258779Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"client":"2","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.106292465Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"io task","client":"0","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.1063251Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"io task","client":"2","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.106356796Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"ds_connection connected from Ok(127.0.0.1:64332)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.106390522Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"io task","client":"0","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"ds_connection connected from Ok(127.0.0.1:33659)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.106421828Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"io task","client":"2","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"ds_connection connected from Ok(127.0.0.1:53282)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.106452664Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.107670881Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"client":"2","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.107707026Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"client":"0","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.108291433Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.108329158Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.108883728Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"client":"1","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"[0]R flush_numbers[0..12]: [303, 312, 312, 291, 312, 296, 311, 312, 312, 299, 312, 312]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109385235Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"downstairs","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"[0]R generation[0..12]: [171, 175, 175, 165, 175, 168, 174, 175, 175, 170, 175, 175]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109419491Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"downstairs","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109442068Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"downstairs","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"[1]R flush_numbers[0..12]: [303, 312, 312, 291, 312, 296, 311, 312, 312, 299, 312, 312]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109465325Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"downstairs","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"[1]R generation[0..12]: [171, 175, 175, 165, 175, 168, 174, 175, 175, 170, 175, 175]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109485692Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"downstairs","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.10950684Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"downstairs","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"[2]R flush_numbers[0..12]: [303, 312, 312, 291, 312, 296, 311, 312, 312, 299, 312, 312]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109527167Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"downstairs","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"[2]R generation[0..12]: [171, 175, 175, 165, 175, 168, 174, 175, 175, 170, 175, 175]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109546715Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"downstairs","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109566292Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"downstairs","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"Max found gen is 176","v":Wait for a query_work_queue command to finish before sending IO 0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109589769Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"downstairs","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"Generation requested: 176 >= found:176","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109606937Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"downstairs","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"Next flush: 313","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109623055Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"downstairs","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109638873Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"":"downstairs","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109655771Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109670899Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"client":"0","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109687707Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"client":"1","":"downstairs","session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109707514Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"client":"2","":"downstairs","session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109726592Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.10974304Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"c298541d-e00b-49cc-8c0d-ec55a7f5e5df is now active with session: f554a9d6-a5f0-48f2-8ac3-253aa917e685","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109760798Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109776836Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910,"session_id":"f554a9d6-a5f0-48f2-8ac3-253aa917e685"} {"msg":"The guest has finished waiting for activation with:176","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109793024Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910} {"msg":"Activated sub_volume c298541d-e00b-49cc-8c0d-ec55a7f5e5df","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:13.109807012Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 331 len 8192 data: 11 11 02/30 Read block 261 len 20480 03/30 Write block 351 len 32768 data: 14 14 18 16 15 13 12 9 04/30 Read block 462 len 36864 05/30 Read block 186 len 16384 06/30 Read block 340 len 32768 07/30 Read block 192 len 16384 08/30 Write block 495 len 12288 data: 6 6 10 09/30 Read block 531 len 8192 10/30 Write block 148 len 36864 data: 7 7 6 6 7 11 13 14 13 11/30 Read block 512 len 32768 12/30 Read block 440 len 36864 13/30 Read block 149 len 4096 14/30 Flush 15/30 Write block 30 len 36864 data: 11 12 13 15 17 20 21 20 24 16/30 Flush 17/30 Write block 406 len 12288 data: 13 13 14 18/30 Read block 245 len 40960 19/30 Write block 524 len 20480 data: 9 12 11 11 12 20/30 Write block 584 len 16384 data: 10 15 14 15 21/30 Read block 194 len 4096 22/30 Write block 516 len 28672 data: 14 15 14 12 11 10 8 23/30 Write block 522 len 40960 data: 9 7 10 13 12 12 13 11 12 15 24/30 Read block 498 len 20480 25/30 Write block 348 len 16384 data: 14 15 17 15 26/30 Flush 27/30 Write block 287 len 12288 data: 13 12 21 28/30 Write block 123 len 40960 data: 16 16 17 16 16 17 20 17 15 20 29/30 Write block 533 len 32768 data: 17 16 17 18 17 17 18 19 30/30 Write block 14 len 8192 data: 8 7 ---------------------------------------------------------------- Crucible gen:176 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Read 5 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Read 9 Done Done Done false 1004 Acked Read 4 Done Done Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Read 4 Done Sent Done false 1007 Acked Write 3 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Write 9 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Read 1 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 9 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Write 4 Done Sent Done false 1020 Acked Read 1 Done Sent Done false 1021 Acked Write 7 Done Sent Done false 1022 Acked Write 10 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Flush 0 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 25 3 31 Done 27 5 27 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) None Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:13.128655997Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1910} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 171 171 171 303 303 303 T F F <--- 1 020-039 176 175 176 314 312 314 F F F <--- 2 040-059 175 175 175 312 312 312 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 175 175 175 312 312 312 F F F 5 100-119 168 168 168 296 296 296 F F F 6 120-139 174 174 174 311 311 311 T F T <--- 7 140-159 176 175 176 313 312 313 F F F <--- 8 160-179 175 175 175 312 312 312 F F F 9 180-199 170 170 170 299 299 299 F F F 10 200-219 175 175 175 312 312 312 F F F 11 220-239 175 175 175 312 312 312 F F F 12 240-259 173 173 173 310 310 310 F F F 13 260-279 172 172 172 308 308 308 F F F 14 280-299 175 175 175 312 312 312 T F T <--- 15 300-319 170 170 170 301 301 301 F F F 16 320-339 176 172 176 313 304 313 F T F <--- 17 340-359 176 172 176 315 307 315 F T F <--- 18 360-379 172 172 172 308 308 308 F F F 19 380-399 172 172 172 308 308 308 F F F 20 400-419 176 172 176 315 308 315 F F F <--- 21 420-439 175 175 175 312 312 312 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 175 175 175 312 312 312 F F F 24 480-499 176 171 176 313 303 313 F F F <--- 25 500-519 176 170 176 315 298 315 F F F <--- 26 520-539 176 175 176 315 312 315 T F T <--- 27 540-559 175 175 175 312 312 312 T F T <--- 28 560-579 173 173 173 310 310 310 F F F 29 580-599 176 175 176 315 312 315 F F F <--- Max gen: 176, Max flush: 315 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 88 Verifying data now No change after verify Loop: 88 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 316 316 316 F F F 1 020-039 176 176 176 314 314 314 F F F 2 040-059 175 175 175 312 312 312 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 175 175 175 312 312 312 F F F 5 100-119 168 168 168 296 296 296 F F F 6 120-139 177 177 177 316 316 316 F F F 7 140-159 176 176 176 313 313 313 F F F 8 160-179 175 175 175 312 312 312 F F F 9 180-199 170 170 170 299 299 299 F F F 10 200-219 175 175 175 312 312 312 F F F 11 220-239 175 175 175 312 312 312 F F F 12 240-259 173 173 173 310 310 310 F F F 13 260-279 172 172 172 308 308 308 F F F 14 280-299 177 177 177 316 316 316 F F F 15 300-319 170 170 170 301 301 301 F F F 16 320-339 176 176 176 313 313 313 F F F 17 340-359 176 176 176 315 315 315 F F F 18 360-379 172 172 172 308 308 308 F F F 19 380-399 172 172 172 308 308 308 F F F 20 400-419 176 176 176 315 315 315 F F F 21 420-439 175 175 175 312 312 312 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 175 175 175 312 312 312 F F F 24 480-499 176 176 176 313 313 313 F F F 25 500-519 176 176 176 315 315 315 F F F 26 520-539 177 177 177 316 316 316 F F F 27 540-559 177 177 177 316 316 316 F F F 28 560-579 173 173 173 310 310 310 F F F 29 580-599 176 176 176 315 315 315 F F F Max gen: 177, Max flush: 316 Begin loop 89 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.53762119Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:17.581700697Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.582112575Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.613840128Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.614400417Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.61445453Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.614473978Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.614502424Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918} {"msg":"Crucible 06828f9c-daed-4180-87fb-f1e660c20e9f has session id: 8d6a1a24-c755-4d53-a04d-f6bcca326df4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615025598Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"Upstairs opts: Upstairs UUID: 06828f9c-daed-4180-87fb-f1e660c20e9f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615079522Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"Crucible stats registered with UUID: 06828f9c-daed-4180-87fb-f1e660c20e9f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615104868Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:17.615125406Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615326311Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"io task","client":"0","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615369655Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"io task","client":"1","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615392972Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615537304Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"task":"crutest"} {"msg":"The guest has requested activation with gen:178","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615556652Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918} {"msg":"06828f9c-daed-4180-87fb-f1e660c20e9f active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615621594Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615640941Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"client":"0","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615663138Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"client":"1","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615692254Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"client":"2","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.61572588Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615759166Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"io task","client":"0","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615796221Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"io task","client":"1","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"ds_connection connected from Ok(127.0.0.1:38311)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615832177Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"ds_connection connected from Ok(127.0.0.1:45262)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615868782Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"io task","client":"0","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"ds_connection connected from Ok(127.0.0.1:61140)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.615904068Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"io task","client":"1","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617094908Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"client":"2","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617137862Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"client":"1","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617177278Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"client":"0","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617686943Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617716999Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"[0]R flush_numbers[0..12]: [316, 314, 312, 291, 312, 296, 316, 313, 312, 299, 312, 312]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617736637Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"downstairs","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"[0]R generation[0..12]: [177, 176, 175, 165, 175, 168, 177, 176, 175, 170, 175, 175]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617756734Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"downstairs","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617776842Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"downstairs","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"[1]R flush_numbers[0..12]: [316, 314, 312, 291, 312, 296, 316, 313, 312, 299, 312, 312]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.6177961Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"downstairs","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"[1]R generation[0..12]: [177, 176, 175, 165, 175, 168, 177, 176, 175, 170, 175, 175]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617815187Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"downstairs","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617834895Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"downstairs","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"[2]R flush_numbers[0..12]: [316, 314, 312, 291, 312, 296, 316, 313, 312, 299, 312, 312]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617853922Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"downstairs","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"[2]R generation[0..12]: [177, 176, 175, 165, 175, 168, 177, 176, 175, 170, 175, 175]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.61787589Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"downstairs","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617896007Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"downstairs","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"Max found gen is 178","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617912855Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"downstairs","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"Generation requested: 178 >= found:178","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617937452Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"downstairs","":"downstairs","session_idWait for a query_work_queue command to finish before sending IO ":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"Next flush: 317","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617969658Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"downstairs","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.617997234Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"":"downstairs","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.618017162Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.618043668Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"client":"0","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.618064716Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"client":"1","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.618081384Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"client":"2","":"downstairs","session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.618098321Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.618123838Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"06828f9c-daed-4180-87fb-f1e660c20e9f is now active with session: 8d6a1a24-c755-4d53-a04d-f6bcca326df4","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.618148025Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.618163593Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918,"session_id":"8d6a1a24-c755-4d53-a04d-f6bcca326df4"} {"msg":"The guest has finished waiting for activation with:178","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.618181781Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918} {"msg":"Activated sub_volume 06828f9c-daed-4180-87fb-f1e660c20e9f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:17.618205028Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 430 len 40960 data: 16 13 12 12 16 13 18 17 15 17 02/30 Write block 255 len 32768 data: 12 11 11 13 7 7 7 7 03/30 Write block 431 len 32768 data: 14 13 13 17 14 19 18 16 04/30 Write block 385 len 36864 data: 15 12 11 12 13 14 14 17 16 05/30 Write block 113 len 16384 data: 15 19 16 22 06/30 Read block 564 len 12288 07/30 Write block 420 len 28672 data: 17 13 15 13 14 15 15 08/30 Read block 186 len 16384 09/30 Read block 61 len 24576 10/30 Read block 554 len 40960 11/30 Write block 332 len 4096 data: 12 12/30 Read block 116 len 40960 13/30 Write block 582 len 12288 data: 10 10 11 14/30 Write block 474 len 24576 data: 14 15 13 12 16 12 15/30 Flush 16/30 Write block 56 len 8192 data: 9 10 17/30 Read block 574 len 16384 18/30 Write block 108 len 4096 data: 15 19/30 Write block 133 len 28672 data: 21 19 16 15 15 15 13 20/30 Write block 188 len 8192 data: 15 13 21/30 Read block 115 len 32768 22/30 Read block 49 len 24576 23/30 Write block 157 len 16384 data: 14 12 14 14 24/30 Read block 87 len 12288 25/30 Flush 26/30 Write block 560 len 4096 data: 9 27/30 Write block 423 len 16384 data: 14 15 16 16 28/30 Read block 503 len 20480 29/30 Write block 295 len 32768 data: 17 15 13 15 14 10 11 12 30/30 Read block 337 len 32768 ---------------------------------------------------------------- Crucible gen:178 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Write 9 Done Done Done false 1004 Acked Write 4 Done Done Done false 1005 Acked Read 3 Sent Done Done false 1006 Acked Write 7 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Read 10 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Write 6 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Write 7 Sent Done Done false 1019 Acked Write 2 Sent Done Done false 1020 Acked Read 8 Sent Done Done false 1021 Acked Read 6 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Write 8 Sent Done Done false 1029 Acked Read 8 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 0 1 26 Done 5 30 29 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:17.637361824Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1918} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 316 316 316 F F F 1 020-039 176 176 176 314 314 314 F F F 2 040-059 175 178 178 312 318 318 F F F <--- 3 060-079 165 165 165 291 291 291 F F F 4 080-099 175 175 175 312 312 312 F F F 5 100-119 168 178 178 296 318 318 T F F <--- 6 120-139 177 178 178 316 318 318 F F F <--- 7 140-159 176 178 178 313 318 318 F F F <--- 8 160-179 175 178 178 312 318 318 F F F <--- 9 180-199 170 178 178 299 318 318 F F F <--- 10 200-219 175 175 175 312 312 312 F F F 11 220-239 175 175 175 312 312 312 F F F 12 240-259 173 178 178 310 317 317 T F F <--- 13 260-279 172 178 178 308 317 317 T F F <--- 14 280-299 177 177 177 316 316 316 F T T <--- 15 300-319 170 170 170 301 301 301 F T T <--- 16 320-339 176 178 178 313 317 317 F F F <--- 17 340-359 176 176 176 315 315 315 F F F 18 360-379 172 172 172 308 308 308 F F F 19 380-399 172 178 178 308 317 317 T F F <--- 20 400-419 176 176 176 315 315 315 F F F 21 420-439 175 178 178 312 317 317 T T T <--- 22 440-459 175 175 175 312 312 312 F F F 23 460-479 175 178 178 312 317 317 F F F <--- 24 480-499 176 176 176 313 313 313 F F F 25 500-519 176 176 176 315 315 315 F F F 26 520-539 177 177 177 316 316 316 F F F 27 540-559 177 177 177 316 316 316 F F F 28 560-579 173 173 173 310 310 310 F T T <--- 29 580-599 176 178 178 315 317 317 F F F <--- Max gen: 178, Max flush: 318 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 89 Verifying data now No change after verify Loop: 89 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 316 316 316 F F F 1 020-039 176 176 176 314 314 314 F F F 2 040-059 178 178 178 318 318 318 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 175 175 175 312 312 312 F F F 5 100-119 178 178 178 318 318 318 F F F 6 120-139 178 178 178 318 318 318 F F F 7 140-159 178 178 178 318 318 318 F F F 8 160-179 178 178 178 318 318 318 F F F 9 180-199 178 178 178 318 318 318 F F F 10 200-219 175 175 175 312 312 312 F F F 11 220-239 175 175 175 312 312 312 F F F 12 240-259 178 178 178 317 317 317 F F F 13 260-279 178 178 178 317 317 317 F F F 14 280-299 179 179 179 319 319 319 F F F 15 300-319 179 179 179 319 319 319 F F F 16 320-339 178 178 178 317 317 317 F F F 17 340-359 176 176 176 315 315 315 F F F 18 360-379 172 172 172 308 308 308 F F F 19 380-399 178 178 178 317 317 317 F F F 20 400-419 176 176 176 315 315 315 F F F 21 420-439 179 179 179 319 319 319 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 178 178 178 317 317 317 F F F 24 480-499 176 176 176 313 313 313 F F F 25 500-519 176 176 176 315 315 315 F F F 26 520-539 177 177 177 316 316 316 F F F 27 540-559 177 177 177 316 316 316 F F F 28 560-579 179 179 179 319 319 319 F F F 29 580-599 178 178 178 317 317 317 F F F Max gen: 179, Max flush: 319 Begin loop 90 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.087458806Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:22.132224796Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.132609577Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.164629313Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.165195532Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.165244046Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.165260983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.165283051Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926} {"msg":"Crucible cdad89fa-ae4e-48e1-ad09-2053f62094a2 has session id: 4a1949ce-6c6b-4c1a-8584-b22b711b3247","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.165891464Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"Upstairs opts: Upstairs UUID: cdad89fa-ae4e-48e1-ad09-2053f62094a2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.165947517Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"Crucible stats registered with UUID: cdad89fa-ae4e-48e1-ad09-2053f62094a2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.165977923Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:22.16600562Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.166223032Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.166263787Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"io task","client":"1","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.166284534Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"io task","client":"2","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.166454153Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"task":"crutest"} {"msg":"The guest has requested activation with gen:180","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.16647511Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926} {"msg":"cdad89fa-ae4e-48e1-ad09-2053f62094a2 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.166541442Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.16656145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"client":"0","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.166578617Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"client":"1","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.166595345Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"client":"2","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.166611963Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"io task","client":"2","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.166629211Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.166646529Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"io task","client":"1","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"ds_connection connected from Ok(127.0.0.1:50956)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.166848903Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"io task","client":"1","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"ds_connection connected from Ok(127.0.0.1:50938)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.166896328Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"io task","client":"2","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"ds_connection connected from Ok(127.0.0.1:51716)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.166933733Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.168237928Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"client":"1","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.168272394Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"client":"0","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.168298731Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"client":"2","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.168871239Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.168900445Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"[0]R flush_numbers[0..12]: [316, 314, 318, 291, 312, 318, 318, 318, 318, 318, 312, 312]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.168919053Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"downstairs","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"[0]R generation[0..12]: [177, 176, 178, 165, 175, 178, 178, 178, 178, 178, 175, 175]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.16893868Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"downstairs","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.168958668Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"downstairs","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"[1]R flush_numbers[0..12]: [316, 314, 318, 291, 312, 318, 318, 318, 318, 318, 312, 312]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.168977815Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"downstairs","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"[1]R generation[0..12]: [177, 176, 178, 165, 175, 178, 178, 178, 178, 178, 175, 175]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.168996943Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"downstairs","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.16901648Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"downstairs","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"[2]R flush_numbers[0..12]: [316, 314, 318, 291, 312, 318, 318, 318, 318, 318, 312, 312]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169035258Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"downstairs","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"[2]R generation[0..12]: [177, 176, 178, 165, 175, 178, 178, 178, 178, 178, 175, 175]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169054036Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"downstairs","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169073653Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"downstairs","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"Max found gen is 180","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169089931Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"downstairs","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} Wait for a query_work_queue command to finish before sending IO {"msg":"Generation requested: 180 >= found:180","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169112648Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"downstairs","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"Next flush: 320","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169133196Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"downstairs","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169149753Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"":"downstairs","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169166841Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169194738Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"client":"0","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169212156Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"client":"1","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169229034Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"client":"2","":"downstairs","session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169246241Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169262579Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"cdad89fa-ae4e-48e1-ad09-2053f62094a2 is now active with session: 4a1949ce-6c6b-4c1a-8584-b22b711b3247","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169280437Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169295745Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926,"session_id":"4a1949ce-6c6b-4c1a-8584-b22b711b3247"} {"msg":"The guest has finished waiting for activation with:180","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169311743Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926} {"msg":"Activated sub_volume cdad89fa-ae4e-48e1-ad09-2053f62094a2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:22.169325541Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Flush 02/30 Write block 263 len 4096 data: 6 03/30 Read block 373 len 16384 04/30 Read block 296 len 28672 05/30 Flush 06/30 Flush 07/30 Flush 08/30 Write block 474 len 28672 data: 15 16 14 13 17 13 12 09/30 Read block 21 len 28672 10/30 Read block 106 len 4096 11/30 Write block 100 len 20480 data: 11 11 12 10 11 12/30 Read block 138 len 36864 13/30 Flush 14/30 Write block 511 len 4096 data: 12 15/30 Read block 504 len 4096 16/30 Read block 464 len 28672 17/30 Write block 556 len 8192 data: 12 13 18/30 Write block 332 len 8192 data: 13 15 19/30 Write block 512 len 4096 data: 14 20/30 Read block 108 len 40960 21/30 Read block 582 len 4096 22/30 Write block 236 len 8192 data: 13 13 23/30 Write block 518 len 20480 data: 15 13 12 11 10 24/30 Read block 352 len 24576 25/30 Write block 510 len 4096 data: 13 26/30 Read block 96 len 28672 27/30 Read block 3 len 12288 28/30 Read block 552 len 32768 29/30 Write block 374 len 16384 data: 8 7 7 10 30/30 Write block 140 len 20480 data: 14 15 15 14 13 ---------------------------------------------------------------- Crucible gen:180 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Flush 0 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 7 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Read 1 Sent Done Done false 1010 Acked Write 5 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 1 Sent Done Done false 1014 Acked Read 1 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Write 2 Sent Done Done false 1017 Acked Write 2 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Write 5 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Write 1 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Read 3 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 2 2 29 Done 0 23 23 46 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1004)) Some(JobId(1012)) Some(JobId(1012)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:22.186184487Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1926} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 316 316 316 F F F 1 020-039 176 176 176 314 314 314 F F F 2 040-059 178 178 178 318 318 318 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 175 175 175 312 312 312 F F F 5 100-119 178 180 180 318 324 324 F F F <--- 6 120-139 178 178 178 318 318 318 F F F 7 140-159 178 178 178 318 318 318 F T T <--- 8 160-179 178 178 178 318 318 318 F F F 9 180-199 178 178 178 318 318 318 F F F 10 200-219 175 175 175 312 312 312 F F F 11 220-239 175 175 175 312 312 312 F T T <--- 12 240-259 178 178 178 317 317 317 F F F 13 260-279 180 180 180 321 321 321 F F F 14 280-299 179 179 179 319 319 319 F F F 15 300-319 179 179 179 319 319 319 F F F 16 320-339 178 178 178 317 317 317 F T T <--- 17 340-359 176 176 176 315 315 315 F F F 18 360-379 172 172 172 308 308 308 F T T <--- 19 380-399 178 178 178 317 317 317 F F F 20 400-419 176 176 176 315 315 315 F F F 21 420-439 179 179 179 319 319 319 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 178 180 180 317 324 324 F F F <--- 24 480-499 176 180 180 313 324 324 F F F <--- 25 500-519 176 176 176 315 315 315 F T T <--- 26 520-539 177 177 177 316 316 316 F T T <--- 27 540-559 177 177 177 316 316 316 F T T <--- 28 560-579 179 179 179 319 319 319 F F F 29 580-599 178 178 178 317 317 317 F F F Max gen: 180, Max flush: 324 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 90 Verifying data now No change after verify Loop: 90 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 316 316 316 F F F 1 020-039 176 176 176 314 314 314 F F F 2 040-059 178 178 178 318 318 318 F F F 3 060-079 165 165 165 291 291 291 F F F 4 080-099 175 175 175 312 312 312 F F F 5 100-119 180 180 180 324 324 324 F F F 6 120-139 178 178 178 318 318 318 F F F 7 140-159 181 181 181 325 325 325 F F F 8 160-179 178 178 178 318 318 318 F F F 9 180-199 178 178 178 318 318 318 F F F 10 200-219 175 175 175 312 312 312 F F F 11 220-239 181 181 181 325 325 325 F F F 12 240-259 178 178 178 317 317 317 F F F 13 260-279 180 180 180 321 321 321 F F F 14 280-299 179 179 179 319 319 319 F F F 15 300-319 179 179 179 319 319 319 F F F 16 320-339 181 181 181 325 325 325 F F F 17 340-359 176 176 176 315 315 315 F F F 18 360-379 181 181 181 325 325 325 F F F 19 380-399 178 178 178 317 317 317 F F F 20 400-419 176 176 176 315 315 315 F F F 21 420-439 179 179 179 319 319 319 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 180 180 180 324 324 324 F F F 24 480-499 180 180 180 324 324 324 F F F 25 500-519 181 181 181 325 325 325 F F F 26 520-539 181 181 181 325 325 325 F F F 27 540-559 181 181 181 325 325 325 F F F 28 560-579 179 179 179 319 319 319 F F F 29 580-599 178 178 178 317 317 317 F F F Max gen: 181, Max flush: 325 Begin loop 91 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.613520566Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:26.65817857Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.658564731Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.690431346Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.691021532Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.691068896Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.691088614Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.69111202Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934} {"msg":"Crucible f435a98c-7292-4d64-aaca-98ec761401b5 has session id: 6f1a445f-d217-4219-810c-b5c5a8b8c8ca","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.691691338Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"Upstairs opts: Upstairs UUID: f435a98c-7292-4d64-aaca-98ec761401b5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.691764568Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"Crucible stats registered with UUID: f435a98c-7292-4d64-aaca-98ec761401b5","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.691794395Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:26.691815672Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.69198878Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692044773Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"io task","client":"0","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692085138Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"io task","client":"1","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692292822Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"task":"crutest"} {"msg":"The guest has requested activation with gen:182","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692321538Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934} {"msg":"f435a98c-7292-4d64-aaca-98ec761401b5 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692409977Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692431684Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"client":"0","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692454891Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"client":"1","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692485627Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"client":"2","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692521913Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692557389Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"io task","client":"0","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692580666Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"io task","client":"1","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"ds_connection connected from Ok(127.0.0.1:43062)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692655826Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"io task","client":"1","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"ds_connection connected from Ok(127.0.0.1:59258)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692907384Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"io task","client":"0","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"ds_connection connected from Ok(127.0.0.1:61938)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.692952759Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.693900689Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"client":"1","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.693935765Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"client":"2","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694188743Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"client":"0","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694479256Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694563456Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"[0]R flush_numbers[0..12]: [316, 314, 318, 291, 312, 324, 318, 325, 318, 318, 312, 325]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694666343Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"downstairs","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"[0]R generation[0..12]: [177, 176, 178, 165, 175, 180, 178, 181, 178, 178, 175, 181]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.69469077Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"downstairs","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694711127Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"downstairs","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"[1]R flush_numbers[0..12]: [316, 314, 318, 291, 312, 324, 318, 325, 318, 318, 312, 325]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694730275Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"downstairs","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"[1]R generation[0..12]: [177, 176, 178, 165, 175, 180, 178, 181, 178, 178, 175, 181]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694752172Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"downstairs","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694772439Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"downstairs","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"[2]R flush_numbers[0..12]: [316, 314, 318, 291, 312, 324, 318, 325, 318, 318, 312, 325]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694791337Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"downstairs","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"[2]R generation[0..12]: [177, 176, 178, 165, 175, 180, 178, 181, 178, 178, 175, 181]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694810145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"downstairs","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694829822Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"downstairs","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"Max found gen is 182","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.69484609Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"Wait for a query_work_queue command to finish before sending IO downstairs","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"Generation requested: 182 >= found:182","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694869647Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"downstairs","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"Next flush: 326","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694885695Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"downstairs","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694901773Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"":"downstairs","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694918731Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694933899Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"client":"0","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694950637Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"client":"1","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694967285Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"client":"2","":"downstairs","session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.694984273Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.69500225Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"f435a98c-7292-4d64-aaca-98ec761401b5 is now active with session: 6f1a445f-d217-4219-810c-b5c5a8b8c8ca","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.695020398Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"session_id":"6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.695035606Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 6f1a445f-d217-4219-810c-b5c5a8b8c8ca"} {"msg":"The guest has finished waiting for activation with:182","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.695055244Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934} {"msg":"Activated sub_volume f435a98c-7292-4d64-aaca-98ec761401b5","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:26.695069412Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 109 len 32768 data: 15 18 18 14 16 20 17 23 02/30 Flush 03/30 Write block 543 len 40960 data: 19 19 18 15 15 16 14 12 14 14 04/30 Read block 329 len 36864 05/30 Write block 393 len 36864 data: 17 17 17 13 13 12 13 13 15 06/30 Read block 284 len 28672 07/30 Flush 08/30 Write block 53 len 40960 data: 12 13 9 10 11 12 9 9 8 6 09/30 Read block 597 len 12288 10/30 Write block 326 len 28672 data: 13 13 13 11 11 12 14 11/30 Read block 568 len 24576 12/30 Write block 353 len 4096 data: 19 13/30 Write block 394 len 32768 data: 18 18 14 14 13 14 14 16 14/30 Read block 277 len 24576 15/30 Read block 77 len 4096 16/30 Write block 373 len 28672 data: 8 9 8 8 11 11 12 17/30 Read block 44 len 16384 18/30 Read block 31 len 12288 19/30 Write block 189 len 4096 data: 14 20/30 Write block 425 len 28672 data: 17 17 13 15 15 17 15 21/30 Write block 579 len 36864 data: 14 10 10 11 11 12 16 15 16 22/30 Read block 341 len 40960 23/30 Read block 214 len 36864 24/30 Write block 580 len 36864 data: 11 11 12 12 13 17 16 17 20 25/30 Read block 7 len 40960 26/30 Write block 475 len 12288 data: 17 15 14 27/30 Read block 275 len 24576 28/30 Write block 364 len 20480 data: 9 10 10 10 12 29/30 Write block 80 len 24576 data: 14 14 16 14 15 11 30/30 Write block 236 len 32768 data: 14 14 15 16 15 16 17 20 ---------------------------------------------------------------- Crucible gen:182 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 10 Sent Done Done false 1003 Acked Read 9 Sent Done Done false 1004 Acked Write 9 Sent Done Done false 1005 Acked Read 7 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 10 Sent Done Done false 1008 Acked Read 3 Sent Done Done false 1009 Acked Write 7 Sent Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Write 1 Sent Done Done false 1012 Acked Write 8 Sent Done Done false 1013 Acked Read 6 Sent Done Done false 1014 Acked Read 1 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Read 4 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Read 10 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Write 9 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Read 6 Sent Done Done false 1027 Acked Write 5 Sent Done Done false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1006)) Some(JobId(1006)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:26.71603556Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1934} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 316 316 316 F F F 1 020-039 176 176 176 314 314 314 F F F 2 040-059 178 178 178 318 318 318 F T T <--- 3 060-079 165 165 165 291 291 291 F T T <--- 4 080-099 175 175 175 312 312 312 F T T <--- 5 100-119 180 182 182 324 326 326 F F F <--- 6 120-139 178 178 178 318 318 318 F F F 7 140-159 181 181 181 325 325 325 F F F 8 160-179 178 178 178 318 318 318 F F F 9 180-199 178 178 178 318 318 318 F T T <--- 10 200-219 175 175 175 312 312 312 F F F 11 220-239 181 181 181 325 325 325 F T T <--- 12 240-259 178 178 178 317 317 317 F T T <--- 13 260-279 180 180 180 321 321 321 F F F 14 280-299 179 179 179 319 319 319 F F F 15 300-319 179 179 179 319 319 319 F F F 16 320-339 181 181 181 325 325 325 F T T <--- 17 340-359 176 176 176 315 315 315 F T T <--- 18 360-379 181 181 181 325 325 325 F T T <--- 19 380-399 178 182 182 317 327 327 F T T <--- 20 400-419 176 182 182 315 327 327 F T T <--- 21 420-439 179 179 179 319 319 319 F T T <--- 22 440-459 175 175 175 312 312 312 F F F 23 460-479 180 180 180 324 324 324 F T T <--- 24 480-499 180 180 180 324 324 324 F F F 25 500-519 181 181 181 325 325 325 F F F 26 520-539 181 181 181 325 325 325 F F F 27 540-559 181 182 182 325 327 327 F F F <--- 28 560-579 179 179 179 319 319 319 F T T <--- 29 580-599 178 178 178 317 317 317 F T T <--- Max gen: 182, Max flush: 327 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 91 Verifying data now No change after verify Loop: 91 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 316 316 316 F F F 1 020-039 176 176 176 314 314 314 F F F 2 040-059 183 183 183 328 328 328 F F F 3 060-079 183 183 183 328 328 328 F F F 4 080-099 183 183 183 328 328 328 F F F 5 100-119 182 182 182 326 326 326 F F F 6 120-139 178 178 178 318 318 318 F F F 7 140-159 181 181 181 325 325 325 F F F 8 160-179 178 178 178 318 318 318 F F F 9 180-199 183 183 183 328 328 328 F F F 10 200-219 175 175 175 312 312 312 F F F 11 220-239 183 183 183 328 328 328 F F F 12 240-259 183 183 183 328 328 328 F F F 13 260-279 180 180 180 321 321 321 F F F 14 280-299 179 179 179 319 319 319 F F F 15 300-319 179 179 179 319 319 319 F F F 16 320-339 183 183 183 328 328 328 F F F 17 340-359 183 183 183 328 328 328 F F F 18 360-379 183 183 183 328 328 328 F F F 19 380-399 183 183 183 328 328 328 F F F 20 400-419 183 183 183 328 328 328 F F F 21 420-439 183 183 183 328 328 328 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 183 183 183 328 328 328 F F F 24 480-499 180 180 180 324 324 324 F F F 25 500-519 181 181 181 325 325 325 F F F 26 520-539 181 181 181 325 325 325 F F F 27 540-559 182 182 182 327 327 327 F F F 28 560-579 183 183 183 328 328 328 F F F 29 580-599 183 183 183 328 328 328 F F F Max gen: 183, Max flush: 328 Begin loop 92 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.189261626Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:31.234005289Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.23439113Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.266216111Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.266750983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.266809686Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.266828913Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.26685628Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942} {"msg":"Crucible ca731a71-bbf3-4edc-92c2-537ed885764e has session id: 870921d0-8fa1-4bdb-9853-bb6fbb1098c7","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.267455894Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"Upstairs opts: Upstairs UUID: ca731a71-bbf3-4edc-92c2-537ed885764e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.267496819Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"Crucible stats registered with UUID: ca731a71-bbf3-4edc-92c2-537ed885764e","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.267519326Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:31.267536144Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.267767495Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"io task","client":"2","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.26781104Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"io task","client":"0","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.267832467Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.26804926Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"task":"crutest"} {"msg":"The guest has requested activation with gen:184","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.268073237Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942} {"msg":"ca731a71-bbf3-4edc-92c2-537ed885764e active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.268146977Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.268168145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"client":"0","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.268185402Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"client":"1","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.26820223Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"client":"2","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.268218858Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"io task","client":"2","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.268240946Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"io task","client":"0","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.268269582Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"ds_connection connected from Ok(127.0.0.1:44472)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.268379708Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"io task","client":"0","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"ds_connection connected from Ok(127.0.0.1:46838)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.268434251Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"io task","client":"2","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"ds_connection connected from Ok(127.0.0.1:44854)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.268507462Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.269732678Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"client":"1","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.269767293Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"client":"2","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.269788891Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"client":"0","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270328043Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270416312Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"[0]R flush_numbers[0..12]: [316, 314, 328, 328, 328, 326, 318, 325, 318, 328, 312, 328]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270532547Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"downstairs","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"[0]R generation[0..12]: [177, 176, 183, 183, 183, 182, 178, 181, 178, 183, 175, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270556184Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"downstairs","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270576531Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"downstairs","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"[1]R flush_numbers[0..12]: [316, 314, 328, 328, 328, 326, 318, 325, 318, 328, 312, 328]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270595629Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"downstairs","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"[1]R generation[0..12]: [177, 176, 183, 183, 183, 182, 178, 181, 178, 183, 175, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270615016Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"downstairs","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270634594Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"downstairs","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"[2]R flush_numbers[0..12]: [316, 314, 328, 328, 328, 326, 318, 325, 318, 328, 312, 328]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270653391Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"downstairs","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"[2]R generation[0..12]: [177, 176, 183, 183, 183, 182, 178, 181, 178, 183, 175, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270675319Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"downstairs","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270699206Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"downstairs","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"Max found gen is 184","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270719113Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"downstairs","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"Generation requested: 184 >= found:184","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270736521Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"downstairs","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"Next flush: 329","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270752619Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"downstairs","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270768937Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"":"downstairs","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270785645Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"session_id":"Wait for a query_work_queue command to finish before sending IO 870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270808012Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"client":"0","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.27082484Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"client":"1","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270841588Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"client":"2","":"downstairs","session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270858546Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270874594Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"ca731a71-bbf3-4edc-92c2-537ed885764e is now active with session: 870921d0-8fa1-4bdb-9853-bb6fbb1098c7","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270892301Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270907379Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942,"session_id":"870921d0-8fa1-4bdb-9853-bb6fbb1098c7"} {"msg":"The guest has finished waiting for activation with:184","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270923897Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942} {"msg":"Activated sub_volume ca731a71-bbf3-4edc-92c2-537ed885764e","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:31.270937966Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 56 len 16384 data: 11 12 13 10 02/30 Read block 351 len 32768 03/30 Read block 251 len 28672 04/30 Write block 534 len 28672 data: 17 18 19 18 18 19 20 05/30 Read block 439 len 32768 06/30 Read block 573 len 36864 07/30 Read block 229 len 40960 08/30 Read block 84 len 28672 09/30 Read block 560 len 24576 10/30 Read block 399 len 40960 11/30 Read block 175 len 24576 12/30 Read block 439 len 24576 13/30 Write block 287 len 4096 data: 14 14/30 Read block 182 len 16384 15/30 Write block 495 len 28672 data: 7 7 11 11 14 12 14 16/30 Write block 201 len 36864 data: 18 16 17 17 19 15 14 12 10 17/30 Write block 166 len 32768 data: 12 12 13 13 9 9 9 8 18/30 Write block 145 len 12288 data: 12 8 7 19/30 Read block 170 len 12288 20/30 Read block 151 len 28672 21/30 Read block 301 len 28672 22/30 Write block 393 len 16384 data: 18 19 19 15 23/30 Read block 310 len 28672 24/30 Read block 146 len 16384 25/30 Read block 43 len 24576 26/30 Write block 119 len 40960 data: 19 20 15 15 17 17 18 17 17 18 27/30 Read block 473 len 16384 28/30 Write block 366 len 12288 data: 11 11 13 29/30 Read block 475 len 16384 30/30 Write block 162 len 24576 data: 16 13 13 12 13 13 ---------------------------------------------------------------- Crucible gen:184 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 8 Done Done Done false 1002 Acked Read 7 Done Done Done false 1003 Acked Write 7 Done Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Read 10 Sent Done Done false 1007 Acked Read 7 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Read 10 Sent Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Read 6 Sent Done Done false 1012 Acked Write 1 Sent Done Done false 1013 Acked Read 4 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 9 Sent Done Done false 1016 Acked Write 8 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Read 3 Sent Done Done false 1019 Acked Read 7 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Read 4 Sent Done Done false 1024 Acked Read 6 Sent Done Done false 1025 Acked Write 10 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Write 3 Sent Done Done false 1028 Acked Read 4 Sent Done Done false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 1 1 28 Done 4 29 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:31.291387069Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1942} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 316 316 316 F F F 1 020-039 176 176 176 314 314 314 F F F 2 040-059 183 183 183 328 328 328 T T T 3 060-079 183 183 183 328 328 328 F F F 4 080-099 183 183 183 328 328 328 F F F 5 100-119 182 182 182 326 326 326 F T T <--- 6 120-139 178 178 178 318 318 318 F T T <--- 7 140-159 181 181 181 325 325 325 F T T <--- 8 160-179 178 178 178 318 318 318 F T T <--- 9 180-199 183 183 183 328 328 328 F F F 10 200-219 175 175 175 312 312 312 F T T <--- 11 220-239 183 183 183 328 328 328 F F F 12 240-259 183 183 183 328 328 328 F F F 13 260-279 180 180 180 321 321 321 F F F 14 280-299 179 179 179 319 319 319 F T T <--- 15 300-319 179 179 179 319 319 319 F F F 16 320-339 183 183 183 328 328 328 F F F 17 340-359 183 183 183 328 328 328 F F F 18 360-379 183 183 183 328 328 328 F T T <--- 19 380-399 183 183 183 328 328 328 F T T <--- 20 400-419 183 183 183 328 328 328 F F F 21 420-439 183 183 183 328 328 328 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 183 183 183 328 328 328 F F F 24 480-499 180 180 180 324 324 324 F T T <--- 25 500-519 181 181 181 325 325 325 F T T <--- 26 520-539 181 181 181 325 325 325 T T T 27 540-559 182 182 182 327 327 327 T T T 28 560-579 183 183 183 328 328 328 F F F 29 580-599 183 183 183 328 328 328 F F F Max gen: 183, Max flush: 328 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 92 Verifying data now No change after verify Loop: 92 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 316 316 316 F F F 1 020-039 176 176 176 314 314 314 F F F 2 040-059 184 184 184 329 329 329 F F F 3 060-079 183 183 183 328 328 328 F F F 4 080-099 183 183 183 328 328 328 F F F 5 100-119 184 184 184 329 329 329 F F F 6 120-139 184 184 184 329 329 329 F F F 7 140-159 184 184 184 329 329 329 F F F 8 160-179 184 184 184 329 329 329 F F F 9 180-199 183 183 183 328 328 328 F F F 10 200-219 184 184 184 329 329 329 F F F 11 220-239 183 183 183 328 328 328 F F F 12 240-259 183 183 183 328 328 328 F F F 13 260-279 180 180 180 321 321 321 F F F 14 280-299 184 184 184 329 329 329 F F F 15 300-319 179 179 179 319 319 319 F F F 16 320-339 183 183 183 328 328 328 F F F 17 340-359 183 183 183 328 328 328 F F F 18 360-379 184 184 184 329 329 329 F F F 19 380-399 184 184 184 329 329 329 F F F 20 400-419 183 183 183 328 328 328 F F F 21 420-439 183 183 183 328 328 328 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 183 183 183 328 328 328 F F F 24 480-499 184 184 184 329 329 329 F F F 25 500-519 184 184 184 329 329 329 F F F 26 520-539 184 184 184 329 329 329 F F F 27 540-559 184 184 184 329 329 329 F F F 28 560-579 183 183 183 328 328 328 F F F 29 580-599 183 183 183 328 328 328 F F F Max gen: 184, Max flush: 329 Begin loop 93 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.75744045Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.802830301Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.803492978Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.80355221Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.803572348Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.803599914Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950} {"msg":"Crucible 06df9272-62e9-4cf2-b33a-ef4e84629e2e has session id: d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804094342Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"Upstairs opts: Upstairs UUID: 06df9272-62e9-4cf2-b33a-ef4e84629e2e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804163273Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"Crucible stats registered with UUID: 06df9272-62e9-4cf2-b33a-ef4e84629e2e","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804200968Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:35.804233004Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804467825Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"io task","client":"2","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.80450506Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804523858Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"io task","client":"1","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804659081Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"task":"crutest"} {"msg":"The guest has requested activation with gen:186","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804683637Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950} {"msg":"06df9272-62e9-4cf2-b33a-ef4e84629e2e active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804760748Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804780645Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"client":"0","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804797913Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"client":"1","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804814731Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"client":"2","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804878793Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"io task","client":"2","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804917698Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804943415Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"io task","client":"1","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"ds_connection connected from Ok(127.0.0.1:53664)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804965132Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"io task","client":"2","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"ds_connection connected from Ok(127.0.0.1:35089)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.804995288Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"io task","client":"1","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"ds_connection connected from Ok(127.0.0.1:39479)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.805027214Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.80609737Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"client":"1","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.806148923Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"client":"0","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.80673136Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.806772854Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.807236236Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"client":"2","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"[0]R flush_numbers[0..12]: [316, 314, 329, 328, 328, 329, 329, 329, 329, 328, 329, 328]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.807745382Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"downstairs","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"[0]R generation[0..12]: [177, 176, 184, 183, 183, 184, 184, 184, 184, 183, 184, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.807779387Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"downstairs","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.807800455Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"downstairs","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"[1]R flush_numbers[0..12]: [316, 314, 329, 328, 328, 329, 329, 329, 329, 328, 329, 328]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.807820093Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"downstairs","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"[1]R generation[0..12]: [177, 176, 184, 183, 183, 184, 184, 184, 184, 183, 184, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.80783898Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"downstairs","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.807858868Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"downstairs","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"[2]R flush_numbers[0..12]: [316, 314, 329, 328, 328, 329, 329, 329, 329, 328, 329, 328]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.807877955Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"downstairs","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"[2]R generation[0..12]: [177, 176, 184, 183, 183, 184, 184, 184, 184, 183, 184, 183]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.807896813Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"downstairs","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.80791646Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"Max found gen is 185","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.807938867Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"downstairs","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"Generation requested: 186 >= found:185","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.807955955Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"downstairs","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"Next flush: 330","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.807972003Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"downstairs","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.807988121Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"":"downstairs","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.808005119Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.808023487Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"client":"0","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.808044884Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"client":"1","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.80807467Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"client":"2","":"downstairs","session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.808108056Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.808129503Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"06df9272-62e9-4cf2-b33a-ef4e84629e2e is now active with session: d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.80815342Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.808180077Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950,"session_id":"d17dcd57-7dc3-4fcf-afa9-0f8e0e91bc4f"} {"msg":"The guest has finished waiting for activation with:186","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.808210023Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950} {"msg":"Activated sub_volume 06df9272-62e9-4cf2-b33a-ef4e84629e2e","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:35.808225931Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 235 len 40960 data: 14 15 15 16 17 16 17 18 21 21 02/30 Read block 98 len 28672 03/30 Read block 426 len 20480 04/30 Read block 275 len 16384 05/30 Write block 210 len 40960 data: 11 11 10 10 9 8 9 9 9 11 06/30 Write block 144 len 12288 data: 14 13 9 07/30 Read block 58 len 8192 08/30 Write block 550 len 36864 data: 13 15 15 15 14 14 13 14 12 09/30 Read block 333 len 32768 10/30 Read block 509 len 40960 11/30 Write block 201 len 20480 data: 19 17 18 18 20 12/30 Write block 120 len 32768 data: 21 16 16 18 18 19 18 18 13/30 Read block 534 len 16384 14/30 Write block 542 len 36864 data: 22 20 20 19 16 16 17 15 14 15/30 Read block 232 len 16384 16/30 Flush 17/30 Write block 572 len 24576 data: 10 12 10 9 11 13 18/30 Read block 77 len 4096 19/30 Write block 395 len 12288 data: 20 16 15 20/30 Write block 50 len 28672 data: 15 13 13 13 14 10 12 21/30 Write block 52 len 40960 data: 14 14 15 11 13 13 14 11 10 9 22/30 Write block 14 len 20480 data: 9 8 6 6 4 23/30 Write block 300 len 40960 data: 11 12 13 10 10 12 15 17 16 16 24/30 Read block 107 len 16384 25/30 Write block 184 len 8192 data: 16 17 26/30 Read block 177 len 36864 27/30 Read block 199 len 24576 28/30 Read block 436 len 12288 29/30 Write block 294 len 16384 data: 19 18 16 14 30/30 Read block 535 len 32768 ---------------------------------------------------------------- Crucible gen:186 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Sent false 1001 Acked Read 7 Done Done Sent false 1002 Acked Read 5 Done Done Sent false 1003 Acked Read 4 Done Done Sent false 1004 Acked Write 10 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Read 2 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Read 8 Done Done Sent false 1009 Acked Read 10 Done Done Sent false 1010 Acked Write 5 Done Done Sent false 1011 Acked Write 8 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 6 Done Done Sent false 1017 Acked Read 1 Done Done Sent false 1018 Acked Write 3 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Write 10 Done Done Sent false 1023 Acked Read 4 Done Done Sent false 1024 Acked Write 2 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Read 6 Done Done Sent false 1027 Acked Read 3 Done Done Sent false 1028 Acked Write 4 Done Sent Sent false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 2 30 32 Done 30 28 0 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) Some(JobId(1015)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:35.828827276Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1950} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 177 177 177 316 316 316 T T F <--- 1 020-039 176 176 176 314 314 314 F F F 2 040-059 184 184 184 329 329 329 T T F <--- 3 060-079 183 183 183 328 328 328 T T F <--- 4 080-099 183 183 183 328 328 328 F F F 5 100-119 184 184 184 329 329 329 F F F 6 120-139 186 186 184 330 330 329 F F F <--- 7 140-159 186 186 184 330 330 329 F F F <--- 8 160-179 184 184 184 329 329 329 F F F 9 180-199 183 183 183 328 328 328 T T F <--- 10 200-219 186 186 184 330 330 329 F F F <--- 11 220-239 186 186 183 330 330 328 F F F <--- 12 240-259 186 186 183 330 330 328 F F F <--- 13 260-279 180 180 180 321 321 321 F F F 14 280-299 184 184 184 329 329 329 T T F <--- 15 300-319 179 179 179 319 319 319 T T F <--- 16 320-339 183 183 183 328 328 328 F F F 17 340-359 183 183 183 328 328 328 F F F 18 360-379 184 184 184 329 329 329 F F F 19 380-399 184 184 184 329 329 329 T T F <--- 20 400-419 183 183 183 328 328 328 F F F 21 420-439 183 183 183 328 328 328 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 183 183 183 328 328 328 F F F 24 480-499 184 184 184 329 329 329 F F F 25 500-519 184 184 184 329 329 329 F F F 26 520-539 184 184 184 329 329 329 F F F 27 540-559 186 186 184 330 330 329 F F F <--- 28 560-579 183 183 183 328 328 328 T T F <--- 29 580-599 183 183 183 328 328 328 F F F Max gen: 186, Max flush: 330 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 93 Verifying data now No change after verify Loop: 93 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 331 331 331 F F F 1 020-039 176 176 176 314 314 314 F F F 2 040-059 187 187 187 331 331 331 F F F 3 060-079 187 187 187 331 331 331 F F F 4 080-099 183 183 183 328 328 328 F F F 5 100-119 184 184 184 329 329 329 F F F 6 120-139 186 186 186 330 330 330 F F F 7 140-159 186 186 186 330 330 330 F F F 8 160-179 184 184 184 329 329 329 F F F 9 180-199 187 187 187 331 331 331 F F F 10 200-219 186 186 186 330 330 330 F F F 11 220-239 186 186 186 330 330 330 F F F 12 240-259 186 186 186 330 330 330 F F F 13 260-279 180 180 180 321 321 321 F F F 14 280-299 187 187 187 331 331 331 F F F 15 300-319 187 187 187 331 331 331 F F F 16 320-339 183 183 183 328 328 328 F F F 17 340-359 183 183 183 328 328 328 F F F 18 360-379 184 184 184 329 329 329 F F F 19 380-399 187 187 187 331 331 331 F F F 20 400-419 183 183 183 328 328 328 F F F 21 420-439 183 183 183 328 328 328 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 183 183 183 328 328 328 F F F 24 480-499 184 184 184 329 329 329 F F F 25 500-519 184 184 184 329 329 329 F F F 26 520-539 184 184 184 329 329 329 F F F 27 540-559 186 186 186 330 330 330 F F F 28 560-579 187 187 187 331 331 331 F F F 29 580-599 183 183 183 328 328 328 F F F Max gen: 187, Max flush: 331 Begin loop 94 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.241332373Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.287269466Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.287879199Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.287927963Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.287944141Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.287966258Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958} {"msg":"Crucible 68c9ffd6-891d-4f76-b385-4d17af3968d3 has session id: f403ad0f-8ef4-4c09-bf93-d1032ea0d60c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.288614236Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"Upstairs opts: Upstairs UUID: 68c9ffd6-891d-4f76-b385-4d17af3968d3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.288655451Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"Crucible stats registered with UUID: 68c9ffd6-891d-4f76-b385-4d17af3968d3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.288674769Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.288691957Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.288908949Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"io task","client":"0","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.288954764Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.288976761Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"io task","client":"1","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.289165347Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"task":"crutest"} {"msg":"The guest has requested activation with gen:188","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.289201682Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958} {"msg":"68c9ffd6-891d-4f76-b385-4d17af3968d3 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.289283212Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.28930382Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"client":"0","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.289326287Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"client":"1","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.289343595Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"client":"2","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.289360122Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.2893773Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"io task","client":"0","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.289468109Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"io task","client":"1","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"ds_connection connected from Ok(127.0.0.1:38340)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.289510674Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"ds_connection connected from Ok(127.0.0.1:38506)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.289541999Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"io task","client":"0","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"ds_connection connected from Ok(127.0.0.1:40178)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.289573786Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"io task","client":"1","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.290671507Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"client":"1","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.290850625Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"client":"0","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.29112769Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"client":"2","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291309367Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291388037Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"[0]R flush_numbers[0..12]: [331, 314, 331, 331, 328, 329, 330, 330, 329, 331, 330, 330]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291662632Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"downstairs","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"[0]R generation[0..12]: [187, 176, 187, 187, 183, 184, 186, 186, 184, 187, 186, 186]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291688569Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"downstairs","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291708917Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"downstairs","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"[1]R flush_numbers[0..12]: [331, 314, 331, 331, 328, 329, 330, 330, 329, 331, 330, 330]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291728844Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"downstairs","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"[1]R generation[0..12]: [187, 176, 187, 187, 183, 184, 186, 186, 184, 187, 186, 186]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291748101Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"downstairs","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291768009Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"downstairs","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"[2]R flush_numbers[0..12]: [331, 314, 331, 331, 328, 329, 330, 330, 329, 331, 330, 330]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291791556Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"downstairs","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"[2]R generation[0..12]: [187, 176, 187, 187, 183, 184, 186, 186, 184, 187, 186, 186]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291810834Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"downstairs","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291830621Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"downstairs","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"Max found gen is 188","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291846949Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"Generation requested: 188 >= found:188","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291869366Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"downstairs","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"Next flush: 332","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291885444Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"downstairs","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291902372Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"":"downstairs","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.2919194Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291934858Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"client":"0","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291952836Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"client":"1","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291969764Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"client":"2","":"downstairs","session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.291986691Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.292003089Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"68c9ffd6-891d-4f76-b385-4d17af3968d3 is now active with session: f403ad0f-8ef4-4c09-bf93-d1032ea0d60c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.292021197Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.292036435Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"The guest has finished waiting for activation with:188","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.292054403Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958} {"msg":"Activated sub_volume 68c9ffd6-891d-4f76-b385-4d17af3968d3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:40.292068131Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 338 len 32768 data: 16 14 11 8 9 9 10 9 02/30 Read block 217 len 32768 03/30 Write block 472 len 28672 data: 17 17 16 18 16 15 18 04/30 Write block 76 len 32768 data: 13 14 18 16 15 15 17 15 05/30 Flush 06/30 Flush 07/30 Write block 568 len 4096 data: 9 08/30 Flush 09/30 Write block 397 len 20480 data: 16 14 15 15 17 10/30 Read block 330 len 8192 11/30 Write block 158 len 24576 data: 13 15 15 15 17 14 12/30 Write block 12 len 40960 data: 10 8 10 9 7 7 5 7 13 12 13/30 Write block 512 len 40960 data: 15 13 14 15 15 16 16 14 13 12 14/30 Read block 448 len 4096 15/30 Read block 19 len 28672 16/30 Read block 23 len 4096 17/30 Write block 559 len 8192 data: 11 10 18/30 Write block 525 len 40960 data: 14 13 13 14 12 13 16 18 18 18 19/30 Read block 302 len 28672 20/30 Read block 343 len 32768 21/30 Read block 132 len 24576 22/30 Read block 544 len 28672 23/30 Write block 271 len 32768 data: 17 17 15 15 16 13 15 14 24/30 Write block 10 len 40960 data: 11 12 11 9 11 10 8 8 6 8 25/30 Read block 361 len 36864 26/30 Read block 201 len 28672 27/30 Write block 313 len 28672 data: 17 18 15 13 13 13 13 28/30 Write block 72 len 40960 data: 10 7 9 9 14 15 19 17 16 16 29/30 Write block 383 len 32768 data: 12 12 16 13 12 13 14 15 30/30 Write block 153 len 24576 data: 12 14 15 14 15 14 ---------------------------------------------------------------- Crucible gen:188 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 8 Done Done Sent false 1002 Acked Write 7 Done Done Sent false 1003 Acked Write 8 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Write 6 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Write 10 Done Done Sent false 1013 Acked Read 1 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Read 1 Done Done Sent false 1016 Acked Write 2 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Read 8 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Read 7 Done Done Sent false 1022 Acked Write 8 Done Done Sent false 1023 Acked Write 10 Done Done Sent false 1024 Acked Read 9 Done Done Sent false 1025 Acked Read 7 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 29 35 Done 27 27 1 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1007)) Some(JobId(1007)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.311865037Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312223332Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312263567Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312294623Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312323999Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312351396Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312378512Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312405899Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312427076Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312443344Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312462442Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312488879Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312513145Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312534163Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312550311Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312566209Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312585016Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312601414Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312617112Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31263283Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312655607Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312679464Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312695962Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31271158Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312727309Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312742936Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312758534Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312774192Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31278983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312806579Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312822526Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312838564Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312854413Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31287259Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312888618Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312904246Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312921064Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312936922Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31295265Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312968558Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.312984256Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313000134Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313016012Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31303156Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313047358Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313063336Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313079134Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313094932Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.3131106Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313126338Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313144986Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313160724Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313176472Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31319194Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313207848Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313223526Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313239304Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313255092Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31327062Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313286508Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313302106Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313317794Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313333602Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31334938Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313365078Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313381256Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313397314Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313415702Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31343182Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313447668Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313463516Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313479044Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313494742Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31351033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313525958Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313541696Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313557364Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313573032Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.3135885Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313604108Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313619906Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313635874Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313651762Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31366769Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313686308Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313701986Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313717654Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313733422Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31374921Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313764868Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313780566Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313796344Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313811952Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31382765Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313843288Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313859056Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313874944Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313891482Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31390724Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313922638Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313938666Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.313970742Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.31398885Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314004808Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314020786Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314037073Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314052691Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.3140684Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314084957Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314100645Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314116294Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314132081Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314148039Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314163757Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314179426Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314194964Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314210631Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314226129Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314244757Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314260555Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314276213Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314292091Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314307849Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314323597Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 2; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314339385Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 0; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314355033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} {"msg":"IO channel closed for 1; we are hopefully exiting","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:40.314370551Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1958,"session_id":"f403ad0f-8ef4-4c09-bf93-d1032ea0d60c"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 331 331 331 T T F <--- 1 020-039 176 176 176 314 314 314 T T F <--- 2 040-059 187 187 187 331 331 331 F F F 3 060-079 188 188 187 332 332 331 T T F <--- 4 080-099 188 188 183 332 332 328 T T F <--- 5 100-119 184 184 184 329 329 329 F F F 6 120-139 186 186 186 330 330 330 F F F 7 140-159 186 186 186 330 330 330 T T F <--- 8 160-179 184 184 184 329 329 329 T T F <--- 9 180-199 187 187 187 331 331 331 F F F 10 200-219 186 186 186 330 330 330 F F F 11 220-239 186 186 186 330 330 330 F F F 12 240-259 186 186 186 330 330 330 F F F 13 260-279 180 180 180 321 321 321 T T F <--- 14 280-299 187 187 187 331 331 331 F F F 15 300-319 187 187 187 331 331 331 T T F <--- 16 320-339 188 188 183 332 332 328 F F T <--- 17 340-359 188 188 183 332 332 328 F F T <--- 18 360-379 184 184 184 329 329 329 F F F 19 380-399 187 187 187 331 331 331 T T F <--- 20 400-419 183 183 183 328 328 328 T T F <--- 21 420-439 183 183 183 328 328 328 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 188 188 183 332 332 328 F F F <--- 24 480-499 184 184 184 329 329 329 F F F 25 500-519 184 184 184 329 329 329 T T F <--- 26 520-539 184 184 184 329 329 329 T T F <--- 27 540-559 186 186 186 330 330 330 T T F <--- 28 560-579 188 188 187 334 334 331 T T F <--- 29 580-599 183 183 183 328 328 328 F F F Max gen: 188, Max flush: 334 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 94 Verifying data now No change after verify Loop: 94 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 189 189 189 335 335 335 F F F 1 020-039 189 189 189 335 335 335 F F F 2 040-059 187 187 187 331 331 331 F F F 3 060-079 189 189 189 335 335 335 F F F 4 080-099 189 189 189 335 335 335 F F F 5 100-119 184 184 184 329 329 329 F F F 6 120-139 186 186 186 330 330 330 F F F 7 140-159 189 189 189 335 335 335 F F F 8 160-179 189 189 189 335 335 335 F F F 9 180-199 187 187 187 331 331 331 F F F 10 200-219 186 186 186 330 330 330 F F F 11 220-239 186 186 186 330 330 330 F F F 12 240-259 186 186 186 330 330 330 F F F 13 260-279 189 189 189 335 335 335 F F F 14 280-299 187 187 187 331 331 331 F F F 15 300-319 189 189 189 335 335 335 F F F 16 320-339 188 188 188 332 332 332 F F F 17 340-359 188 188 188 332 332 332 F F F 18 360-379 184 184 184 329 329 329 F F F 19 380-399 189 189 189 335 335 335 F F F 20 400-419 189 189 189 335 335 335 F F F 21 420-439 183 183 183 328 328 328 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 188 188 188 332 332 332 F F F 24 480-499 184 184 184 329 329 329 F F F 25 500-519 189 189 189 335 335 335 F F F 26 520-539 189 189 189 335 335 335 F F F 27 540-559 189 189 189 335 335 335 F F F 28 560-579 189 189 189 335 335 335 F F F 29 580-599 183 183 183 328 328 328 F F F Max gen: 189, Max flush: 335 Begin loop 95 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.755358079Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.800586271Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.801214612Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.801261296Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.801277234Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.801299631Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966} {"msg":"Crucible 5c341740-81cf-49af-9902-415d2ba05e7b has session id: 99ff76b4-fb03-4bae-8e3d-9e7870e8d468","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.80186484Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"Upstairs opts: Upstairs UUID: 5c341740-81cf-49af-9902-415d2ba05e7b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.801920343Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"Crucible stats registered with UUID: 5c341740-81cf-49af-9902-415d2ba05e7b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.80194389Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:44.801964097Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802176231Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802221785Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"io task","client":"2","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802246142Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"io task","client":"0","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802469994Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"task":"crutest"} {"msg":"The guest has requested activation with gen:190","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802514638Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966} {"msg":"5c341740-81cf-49af-9902-415d2ba05e7b active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802556913Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.80257459Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"client":"0","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802591438Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"client":"1","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802608266Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"client":"2","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802624694Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"io task","client":"2","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802700685Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802743529Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"io task","client":"0","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"ds_connection connected from Ok(127.0.0.1:61563)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802770906Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"io task","client":"2","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"ds_connection connected from Ok(127.0.0.1:48617)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802801392Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"io task","client":"0","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"ds_connection connected from Ok(127.0.0.1:53167)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.802960612Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.804033757Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"client":"1","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.804080961Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"client":"0","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.804666007Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.804695853Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.804715451Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"client":"2","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"[0]R flush_numbers[0..12]: [335, 335, 331, 335, 335, 329, 330, 335, 335, 331, 330, 330]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805256872Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"downstairs","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"[0]R generation[0..12]: [189, 189, 187, 189, 189, 184, 186, 189, 189, 187, 186, 186]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805290208Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"downstairs","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805311206Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"downstairs","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"[1]R flush_numbers[0..12]: [335, 335, 331, 335, 335, 329, 330, 335, 335, 331, 330, 330]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805330383Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"downstairs","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"[1]R generation[0..12]: [189, 189, 187, 189, 189, 184, 186, 189, 189, 187, 186, 186]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805349521Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"downstairs","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805369358Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"downstairs","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"[2]R flush_numbers[0..12]: [335, 335, 331, 335, 335, 329, 330, 335, 335, 331, 330, 330]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805391815Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"downstairs","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"[2]R generation[0..12]: [189, 189, 187, 189, 189, 184, 186, 189, 189, 187, 186, 186]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805410983Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"downstairs","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.80543404Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"downstairs","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"Max found gen is 190","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805456867Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"downstairs","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"Generation requested: 190 >= found:190","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805474505Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"downstairs","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"Next flush: 336","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805490513Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"downstairs","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805506831Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"":"downstairs","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805523949Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} Wait for a query_work_queue command to finish before sending IO {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805545386Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"client":"0","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805566844Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"client":"1","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805585681Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"client":"2","":"downstairs","session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805602679Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805618727Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"5c341740-81cf-49af-9902-415d2ba05e7b is now active with session: 99ff76b4-fb03-4bae-8e3d-9e7870e8d468","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805636805Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805651953Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966,"session_id":"99ff76b4-fb03-4bae-8e3d-9e7870e8d468"} {"msg":"The guest has finished waiting for activation with:190","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.80567Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966} {"msg":"Activated sub_volume 5c341740-81cf-49af-9902-415d2ba05e7b","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:44.805684019Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 192 len 20480 data: 13 12 12 10 12 02/30 Write block 146 len 40960 data: 10 8 8 8 7 7 8 13 15 16 03/30 Read block 212 len 4096 04/30 Read block 84 len 32768 05/30 Write block 170 len 36864 data: 10 10 10 9 9 8 11 13 15 06/30 Write block 514 len 16384 data: 15 16 16 17 07/30 Read block 388 len 36864 08/30 Write block 252 len 4096 data: 10 09/30 Write block 435 len 4096 data: 15 10/30 Read block 466 len 32768 11/30 Read block 359 len 12288 12/30 Read block 88 len 28672 13/30 Read block 594 len 8192 14/30 Flush 15/30 Write block 306 len 28672 data: 16 18 17 17 15 17 18 16/30 Read block 275 len 8192 17/30 Read block 432 len 12288 18/30 Read block 477 len 12288 19/30 Read block 146 len 8192 20/30 Read block 348 len 4096 21/30 Write block 278 len 20480 data: 15 14 10 11 12 22/30 Write block 483 len 36864 data: 14 14 13 9 10 9 8 5 5 23/30 Read block 315 len 4096 24/30 Write block 550 len 28672 data: 15 16 16 16 15 15 14 25/30 Read block 455 len 40960 26/30 Read block 406 len 12288 27/30 Read block 347 len 12288 28/30 Read block 308 len 8192 29/30 Write block 302 len 20480 data: 14 11 11 13 17 30/30 Read block 72 len 32768 ---------------------------------------------------------------- Crucible gen:190 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Read 8 Done Done Sent false 1004 Acked Write 9 Done Done Sent false 1005 Acked Write 4 Done Done Sent false 1006 Acked Read 9 Done Done Sent false 1007 Acked Write 1 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Read 3 Done Done Sent false 1011 Acked Read 7 Done Done Sent false 1012 Acked Read 2 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Read 3 Done Done Sent false 1018 Acked Read 2 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Write 5 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Read 1 Done Done Sent false 1023 Acked Write 7 Done Done Sent false 1024 Acked Read 10 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Write 5 Done Done Sent false 1029 Acked Read 8 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 27 28 Done 30 29 3 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1013)) Some(JobId(1013)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:44.824840985Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1966} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 189 189 189 335 335 335 F F F 1 020-039 189 189 189 335 335 335 F F F 2 040-059 187 187 187 331 331 331 F F F 3 060-079 189 189 189 335 335 335 F F F 4 080-099 189 189 189 335 335 335 F F F 5 100-119 184 184 184 329 329 329 F F F 6 120-139 186 186 186 330 330 330 F F F 7 140-159 190 190 189 336 336 335 F F T <--- 8 160-179 190 190 189 336 336 335 F F F <--- 9 180-199 190 190 187 336 336 331 F F T <--- 10 200-219 186 186 186 330 330 330 F F F 11 220-239 186 186 186 330 330 330 F F F 12 240-259 190 190 186 336 336 330 F F F <--- 13 260-279 189 189 189 335 335 335 T T F <--- 14 280-299 187 187 187 331 331 331 T T F <--- 15 300-319 189 189 189 335 335 335 T T F <--- 16 320-339 188 188 188 332 332 332 F F F 17 340-359 188 188 188 332 332 332 F F F 18 360-379 184 184 184 329 329 329 F F F 19 380-399 189 189 189 335 335 335 F F F 20 400-419 189 189 189 335 335 335 F F F 21 420-439 190 190 183 336 336 328 F F F <--- 22 440-459 175 175 175 312 312 312 F F F 23 460-479 188 188 188 332 332 332 F F F 24 480-499 184 184 184 329 329 329 T T F <--- 25 500-519 190 190 189 336 336 335 F F F <--- 26 520-539 189 189 189 335 335 335 F F F 27 540-559 189 189 189 335 335 335 T T F <--- 28 560-579 189 189 189 335 335 335 F F F 29 580-599 183 183 183 328 328 328 F F F Max gen: 190, Max flush: 336 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 95 Verifying data now No change after verify Loop: 95 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 189 189 189 335 335 335 F F F 1 020-039 189 189 189 335 335 335 F F F 2 040-059 187 187 187 331 331 331 F F F 3 060-079 189 189 189 335 335 335 F F F 4 080-099 189 189 189 335 335 335 F F F 5 100-119 184 184 184 329 329 329 F F F 6 120-139 186 186 186 330 330 330 F F F 7 140-159 190 190 190 336 336 336 F F F 8 160-179 190 190 190 336 336 336 F F F 9 180-199 190 190 190 336 336 336 F F F 10 200-219 186 186 186 330 330 330 F F F 11 220-239 186 186 186 330 330 330 F F F 12 240-259 190 190 190 336 336 336 F F F 13 260-279 191 191 191 337 337 337 F F F 14 280-299 191 191 191 337 337 337 F F F 15 300-319 191 191 191 337 337 337 F F F 16 320-339 188 188 188 332 332 332 F F F 17 340-359 188 188 188 332 332 332 F F F 18 360-379 184 184 184 329 329 329 F F F 19 380-399 189 189 189 335 335 335 F F F 20 400-419 189 189 189 335 335 335 F F F 21 420-439 190 190 190 336 336 336 F F F 22 440-459 175 175 175 312 312 312 F F F 23 460-479 188 188 188 332 332 332 F F F 24 480-499 191 191 191 337 337 337 F F F 25 500-519 190 190 190 336 336 336 F F F 26 520-539 189 189 189 335 335 335 F F F 27 540-559 191 191 191 337 337 337 F F F 28 560-579 189 189 189 335 335 335 F F F 29 580-599 183 183 183 328 328 328 F F F Max gen: 191, Max flush: 337 Begin loop 96 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.238255248Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:49.282367211Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.282757781Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.314102072Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.31467979Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.314725384Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.314740792Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.314762689Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974} {"msg":"Crucible 3727676b-6696-405e-a151-0c1de688125a has session id: 9e50fbd6-ae20-4cae-a39f-f9772cf1390a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.315292073Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"Upstairs opts: Upstairs UUID: 3727676b-6696-405e-a151-0c1de688125a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.315344546Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"Crucible stats registered with UUID: 3727676b-6696-405e-a151-0c1de688125a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.315366283Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:49.3153949Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.315580646Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"io task","client":"0","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.31562782Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"io task","client":"1","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.315648778Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.315841493Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"task":"crutest"} {"msg":"The guest has requested activation with gen:192","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.31586933Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974} {"msg":"3727676b-6696-405e-a151-0c1de688125a active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.315936631Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.315968257Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"client":"0","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.316000153Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"client":"1","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.316032589Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"client":"2","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.316065015Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.316119918Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"io task","client":"1","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.316150874Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"io task","client":"0","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"ds_connection connected from Ok(127.0.0.1:33302)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.31618605Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"ds_connection connected from Ok(127.0.0.1:41160)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.316234874Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"io task","client":"1","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"ds_connection connected from Ok(127.0.0.1:44813)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.316399243Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"io task","client":"0","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.317275363Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"client":"1","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.317323966Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"client":"2","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.317371551Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"client":"0","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.317892895Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.317988803Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"[0]R flush_numbers[0..12]: [335, 335, 331, 335, 335, 329, 330, 336, 336, 336, 330, 330]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318031757Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"downstairs","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"[0]R generation[0..12]: [189, 189, 187, 189, 189, 184, 186, 190, 190, 190, 186, 186]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318053855Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"downstairs","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318074012Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"downstairs","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"[1]R flush_numbers[0..12]: [335, 335, 331, 335, 335, 329, 330, 336, 336, 336, 330, 330]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.31809314Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"downstairs","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"[1]R generation[0..12]: [189, 189, 187, 189, 189, 184, 186, 190, 190, 190, 186, 186]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318113997Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"downstairs","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318134204Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"downstairs","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"[2]R flush_numbers[0..12]: [335, 335, 331, 335, 335, 329, 330, 336, 336, 336, 330, 330]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318153252Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"downstairs","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"[2]R generation[0..12]: [189, 189, 187, 189, 189, 184, 186, 190, 190, 190, 186, 186]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318177329Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"downstairs","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318210965Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"downstairs","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"Max found gen is 192","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318234622Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"downstairs","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"Wait for a query_work_queue command to finish before sending IO Generation requested: 192 >= found:192","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318260349Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"downstairs","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"Next flush: 338","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318277926Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"downstairs","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318294334Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"":"downstairs","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318311292Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.31832662Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"client":"0","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318343328Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"client":"1","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318359986Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"client":"2","":"downstairs","session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318377014Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318393262Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"3727676b-6696-405e-a151-0c1de688125a is now active with session: 9e50fbd6-ae20-4cae-a39f-f9772cf1390a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.31841119Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318426588Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974,"session_id":"9e50fbd6-ae20-4cae-a39f-f9772cf1390a"} {"msg":"The guest has finished waiting for activation with:192","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318442836Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974} {"msg":"Activated sub_volume 3727676b-6696-405e-a151-0c1de688125a","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:49.318456714Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 479 len 40960 data: 14 13 14 13 15 15 14 10 11 10 02/30 Read block 525 len 36864 03/30 Flush 04/30 Write block 6 len 12288 data: 8 9 10 05/30 Write block 51 len 24576 data: 14 15 15 16 12 14 06/30 Read block 93 len 36864 07/30 Read block 187 len 36864 08/30 Read block 298 len 12288 09/30 Write block 119 len 8192 data: 20 22 10/30 Read block 587 len 12288 11/30 Write block 337 len 12288 data: 17 17 15 12/30 Write block 195 len 40960 data: 11 13 16 16 15 16 20 18 19 19 13/30 Read block 121 len 32768 14/30 Write block 39 len 16384 data: 21 23 24 21 15/30 Read block 555 len 36864 16/30 Write block 231 len 4096 data: 11 17/30 Flush 18/30 Write block 446 len 12288 data: 11 12 15 19/30 Flush 20/30 Write block 428 len 20480 data: 16 16 18 16 14 21/30 Write block 340 len 28672 data: 12 9 10 10 11 10 10 22/30 Write block 557 len 8192 data: 15 13 23/30 Read block 258 len 32768 24/30 Write block 52 len 32768 data: 16 16 17 13 15 14 15 12 25/30 Flush 26/30 Write block 169 len 36864 data: 14 11 11 11 10 10 9 12 14 27/30 Read block 405 len 20480 28/30 Write block 175 len 24576 data: 10 13 15 16 21 23 29/30 Write block 352 len 4096 data: 15 30/30 Read block 24 len 4096 ---------------------------------------------------------------- Crucible gen:192 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 9 Done Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 3 Sent Done Done false 1004 Acked Write 6 Sent Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Read 9 Sent Done Done false 1007 Acked Read 3 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Read 3 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Read 8 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Write 1 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Write 2 Sent Done Done false 1022 Acked Read 8 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 9 Sent Done Done false 1026 Acked Read 5 Sent Done Done false 1027 Acked Write 6 Sent Done Sent false 1028 Acked Write 1 Sent Done Sent false 1029 Acked Read 1 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 3 31 Done 2 30 27 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:49.338546813Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1974} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 189 192 192 335 339 339 F F F <--- 1 020-039 189 192 192 335 339 339 F F F <--- 2 040-059 187 192 192 331 341 341 F F F <--- 3 060-079 189 189 189 335 335 335 F F F 4 080-099 189 189 189 335 335 335 F F F 5 100-119 184 192 192 329 339 339 F F F <--- 6 120-139 186 192 192 330 339 339 F F F <--- 7 140-159 190 190 190 336 336 336 F F F 8 160-179 190 190 190 336 336 336 F T T <--- 9 180-199 190 192 192 336 339 339 F T T <--- 10 200-219 186 192 192 330 339 339 F F F <--- 11 220-239 186 192 192 330 339 339 F F F <--- 12 240-259 190 190 190 336 336 336 F F F 13 260-279 191 191 191 337 337 337 F F F 14 280-299 191 191 191 337 337 337 F F F 15 300-319 191 191 191 337 337 337 F F F 16 320-339 188 192 192 332 339 339 F F F <--- 17 340-359 188 192 192 332 341 341 F T T <--- 18 360-379 184 184 184 329 329 329 F F F 19 380-399 189 189 189 335 335 335 F F F 20 400-419 189 189 189 335 335 335 F F F 21 420-439 190 192 192 336 341 341 F F F <--- 22 440-459 175 192 192 312 340 340 F F F <--- 23 460-479 188 192 192 332 338 338 T F F <--- 24 480-499 191 192 192 337 338 338 T F F <--- 25 500-519 190 190 190 336 336 336 F F F 26 520-539 189 189 189 335 335 335 F F F 27 540-559 191 192 192 337 341 341 F F F <--- 28 560-579 189 189 189 335 335 335 F F F 29 580-599 183 183 183 328 328 328 F F F Max gen: 192, Max flush: 341 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 96 Verifying data now No change after verify Loop: 96 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 339 339 339 F F F 1 020-039 192 192 192 339 339 339 F F F 2 040-059 192 192 192 341 341 341 F F F 3 060-079 189 189 189 335 335 335 F F F 4 080-099 189 189 189 335 335 335 F F F 5 100-119 192 192 192 339 339 339 F F F 6 120-139 192 192 192 339 339 339 F F F 7 140-159 190 190 190 336 336 336 F F F 8 160-179 193 193 193 342 342 342 F F F 9 180-199 193 193 193 342 342 342 F F F 10 200-219 192 192 192 339 339 339 F F F 11 220-239 192 192 192 339 339 339 F F F 12 240-259 190 190 190 336 336 336 F F F 13 260-279 191 191 191 337 337 337 F F F 14 280-299 191 191 191 337 337 337 F F F 15 300-319 191 191 191 337 337 337 F F F 16 320-339 192 192 192 339 339 339 F F F 17 340-359 193 193 193 342 342 342 F F F 18 360-379 184 184 184 329 329 329 F F F 19 380-399 189 189 189 335 335 335 F F F 20 400-419 189 189 189 335 335 335 F F F 21 420-439 192 192 192 341 341 341 F F F 22 440-459 192 192 192 340 340 340 F F F 23 460-479 192 192 192 338 338 338 F F F 24 480-499 192 192 192 338 338 338 F F F 25 500-519 190 190 190 336 336 336 F F F 26 520-539 189 189 189 335 335 335 F F F 27 540-559 192 192 192 341 341 341 F F F 28 560-579 189 189 189 335 335 335 F F F 29 580-599 183 183 183 328 328 328 F F F Max gen: 193, Max flush: 342 Begin loop 97 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.784962977Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.831164297Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.831757842Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.831806336Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.831827363Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.83185007Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982} {"msg":"Crucible a0190604-f248-497d-824f-5c9d1a1844e2 has session id: aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.832389802Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"Upstairs opts: Upstairs UUID: a0190604-f248-497d-824f-5c9d1a1844e2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.832446265Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"Crucible stats registered with UUID: a0190604-f248-497d-824f-5c9d1a1844e2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.832477941Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:53.832506058Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.832701523Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"io task","client":"0","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.832750337Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"io task","client":"1","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.832780763Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"io task","client":"2","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.832996186Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"task":"crutest"} {"msg":"The guest has requested activation with gen:194","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.833032161Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982} {"msg":"a0190604-f248-497d-824f-5c9d1a1844e2 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.833064307Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.833080455Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"client":"0","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.833098693Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"client":"1","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.83311922Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"client":"2","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.833136118Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"io task","client":"2","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.833228247Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"io task","client":"0","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.833266822Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"io task","client":"1","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"ds_connection connected from Ok(127.0.0.1:54321)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.833300698Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"io task","client":"2","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"ds_connection connected from Ok(127.0.0.1:62392)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.833332174Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"io task","client":"1","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"ds_connection connected from Ok(127.0.0.1:33063)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.833366379Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"io task","client":"0","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.834559889Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"client":"1","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.834601204Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"client":"0","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.834639259Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"client":"2","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835214397Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835243243Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"[0]R flush_numbers[0..12]: [339, 339, 341, 335, 335, 339, 339, 336, 342, 342, 339, 339]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.83526383Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"downstairs","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"[0]R generation[0..12]: [192, 192, 192, 189, 189, 192, 192, 190, 193, 193, 192, 192]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835283498Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"downstairs","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835303335Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"downstairs","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"[1]R flush_numbers[0..12]: [339, 339, 341, 335, 335, 339, 339, 336, 342, 342, 339, 339]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835326062Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"downstairs","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"[1]R generation[0..12]: [192, 192, 192, 189, 189, 192, 192, 190, 193, 193, 192, 192]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.83534533Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"downstairs","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835364927Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"downstairs","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"[2]R flush_numbers[0..12]: [339, 339, 341, 335, 335, 339, 339, 336, 342, 342, 339, 339]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835383855Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"downstairs","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"[2]R generation[0..12]: [192, 192, 192, 189, 189, 192, 192, 190, 193, 193, 192, 192]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835402543Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"downstairs","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.83542181Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"downstairs","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"Max found gen is 194","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835438028Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"downstairs","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"Generation requested: 194 >= found:194","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835460135Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"downstairs","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"Next flush: 343","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2025-11-06T20:19:53.835493911Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"downstairs","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835512499Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"":"downstairs","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835529997Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835561343Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"client":"0","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835588059Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"client":"1","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835606127Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"client":"2","":"downstairs","session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835623265Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835647502Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"a0190604-f248-497d-824f-5c9d1a1844e2 is now active with session: aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835675158Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835691686Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982,"session_id":"aa9746e8-0ea1-415d-b4e1-cfb90cd9d86c"} {"msg":"The guest has finished waiting for activation with:194","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835707754Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982} {"msg":"Activated sub_volume a0190604-f248-497d-824f-5c9d1a1844e2","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:53.835729832Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 519 len 20480 data: 15 14 13 11 8 02/30 Read block 338 len 28672 03/30 Flush 04/30 Write block 273 len 28672 data: 16 16 17 14 16 16 15 05/30 Flush 06/30 Write block 252 len 28672 data: 11 11 14 13 12 12 14 07/30 Flush 08/30 Write block 130 len 16384 data: 18 16 21 22 09/30 Read block 464 len 16384 10/30 Write block 245 len 20480 data: 22 19 15 14 15 11/30 Write block 526 len 16384 data: 14 14 15 13 12/30 Write block 424 len 28672 data: 16 18 18 14 17 17 19 13/30 Write block 273 len 20480 data: 17 17 18 15 17 14/30 Write block 499 len 36864 data: 15 13 15 14 14 11 12 13 9 15/30 Read block 581 len 36864 16/30 Read block 76 len 16384 17/30 Write block 403 len 32768 data: 15 14 14 14 14 15 14 13 18/30 Read block 31 len 40960 19/30 Write block 592 len 20480 data: 18 15 14 14 13 20/30 Flush 21/30 Write block 556 len 8192 data: 15 16 22/30 Read block 61 len 40960 23/30 Read block 340 len 36864 24/30 Read block 452 len 4096 25/30 Write block 92 len 40960 data: 16 15 14 17 15 15 14 15 12 12 26/30 Write block 66 len 36864 data: 12 11 13 12 11 11 11 8 10 27/30 Write block 300 len 36864 data: 12 13 15 12 12 14 18 19 18 28/30 Write block 384 len 16384 data: 13 17 14 13 29/30 Write block 400 len 16384 data: 16 18 15 16 30/30 Read block 466 len 20480 ---------------------------------------------------------------- Crucible gen:194 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Sent false 1001 Acked Read 7 Done Done Sent false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 7 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Read 4 Done Done Sent false 1009 Acked Write 5 Done Done Sent false 1010 Acked Write 4 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Write 9 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Read 4 Done Done Sent false 1016 Acked Write 8 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Write 5 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 2 Done Done Sent false 1021 Acked Read 10 Done Done Sent false 1022 Acked Read 9 Done Done Sent false 1023 Acked Read 1 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Write 9 Done Done Sent false 1026 Acked Write 9 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Read 5 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 30 31 Done 30 29 0 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:53.854708281Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1982} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 339 339 339 F F F 1 020-039 192 192 192 339 339 339 F F F 2 040-059 192 192 192 341 341 341 F F F 3 060-079 189 189 189 335 335 335 T T F <--- 4 080-099 189 189 189 335 335 335 T T F <--- 5 100-119 192 192 192 339 339 339 T T F <--- 6 120-139 194 194 192 346 346 339 F F F <--- 7 140-159 190 190 190 336 336 336 F F F 8 160-179 193 193 193 342 342 342 F F F 9 180-199 193 193 193 342 342 342 F F F 10 200-219 192 192 192 339 339 339 F F F 11 220-239 192 192 192 339 339 339 F F F 12 240-259 194 194 190 346 346 336 F F F <--- 13 260-279 194 194 191 346 346 337 F F F <--- 14 280-299 191 191 191 337 337 337 F F F 15 300-319 191 191 191 337 337 337 T T F <--- 16 320-339 192 192 192 339 339 339 F F F 17 340-359 193 193 193 342 342 342 F F F 18 360-379 184 184 184 329 329 329 F F F 19 380-399 189 189 189 335 335 335 T T F <--- 20 400-419 194 194 189 346 346 335 T T F <--- 21 420-439 194 194 192 346 346 341 F F F <--- 22 440-459 192 192 192 340 340 340 F F F 23 460-479 192 192 192 338 338 338 F F F 24 480-499 194 194 192 346 346 338 F F F <--- 25 500-519 194 194 190 346 346 336 F F F <--- 26 520-539 194 194 189 346 346 335 F F F <--- 27 540-559 192 192 192 341 341 341 T T F <--- 28 560-579 189 189 189 335 335 335 F F F 29 580-599 194 194 183 346 346 328 F F F <--- Max gen: 194, Max flush: 346 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 97 Verifying data now No change after verify Loop: 97 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 339 339 339 F F F 1 020-039 192 192 192 339 339 339 F F F 2 040-059 192 192 192 341 341 341 F F F 3 060-079 195 195 195 347 347 347 F F F 4 080-099 195 195 195 347 347 347 F F F 5 100-119 195 195 195 347 347 347 F F F 6 120-139 194 194 194 346 346 346 F F F 7 140-159 190 190 190 336 336 336 F F F 8 160-179 193 193 193 342 342 342 F F F 9 180-199 193 193 193 342 342 342 F F F 10 200-219 192 192 192 339 339 339 F F F 11 220-239 192 192 192 339 339 339 F F F 12 240-259 194 194 194 346 346 346 F F F 13 260-279 194 194 194 346 346 346 F F F 14 280-299 191 191 191 337 337 337 F F F 15 300-319 195 195 195 347 347 347 F F F 16 320-339 192 192 192 339 339 339 F F F 17 340-359 193 193 193 342 342 342 F F F 18 360-379 184 184 184 329 329 329 F F F 19 380-399 195 195 195 347 347 347 F F F 20 400-419 195 195 195 347 347 347 F F F 21 420-439 194 194 194 346 346 346 F F F 22 440-459 192 192 192 340 340 340 F F F 23 460-479 192 192 192 338 338 338 F F F 24 480-499 194 194 194 346 346 346 F F F 25 500-519 194 194 194 346 346 346 F F F 26 520-539 194 194 194 346 346 346 F F F 27 540-559 195 195 195 347 347 347 F F F 28 560-579 189 189 189 335 335 335 F F F 29 580-599 194 194 194 346 346 346 F F F Max gen: 195, Max flush: 347 Begin loop 98 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.26965442Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:58.313751444Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.314147894Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.345607701Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.346148203Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.346195227Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.346211335Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.346234302Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990} {"msg":"Crucible 14c6bb53-24af-4208-b8c9-6ab806662b66 has session id: bef23214-d5ef-40cc-a3b4-7078b66db151","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.346797841Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"Upstairs opts: Upstairs UUID: 14c6bb53-24af-4208-b8c9-6ab806662b66, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.346856753Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"Crucible stats registered with UUID: 14c6bb53-24af-4208-b8c9-6ab806662b66","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.346889429Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:58.346916266Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.347133139Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"io task","client":"0","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.347174033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"io task","client":"1","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.347193961Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"io task","client":"2","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.347421852Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"task":"crutest"} {"msg":"The guest has requested activation with gen:196","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.347445929Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990} {"msg":"14c6bb53-24af-4208-b8c9-6ab806662b66 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.34752136Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.347542247Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"client":"0","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.347559475Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"client":"1","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.347576273Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"client":"2","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.347649884Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"io task","client":"2","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.347695048Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"io task","client":"0","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.347895253Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"io task","client":"1","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"ds_connection connected from Ok(127.0.0.1:45411)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.347930518Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"io task","client":"2","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"ds_connection connected from Ok(127.0.0.1:44616)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.347958925Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"io task","client":"1","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"ds_connection connected from Ok(127.0.0.1:63390)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.34799205Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"io task","client":"0","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349036879Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"client":"1","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349151494Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"client":"0","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349194519Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"client":"2","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349607207Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349703045Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"[0]R flush_numbers[0..12]: [339, 339, 341, 347, 347, 347, 346, 336, 342, 342, 339, 339]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349734861Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"downstairs","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"[0]R generation[0..12]: [192, 192, 192, 195, 195, 195, 194, 190, 193, 193, 192, 192]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349756368Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"downstairs","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349776725Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"downstairs","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"[1]R flush_numbers[0..12]: [339, 339, 341, 347, 347, 347, 346, 336, 342, 342, 339, 339]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349796073Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"downstairs","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"[1]R generation[0..12]: [192, 192, 192, 195, 195, 195, 194, 190, 193, 193, 192, 192]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349815101Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"downstairs","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349835178Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"downstairs","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"[2]R flush_numbers[0..12]: [339, 339, 341, 347, 347, 347, 346, 336, 342, 342, 339, 339]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349854046Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"downstairs","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"[2]R generation[0..12]: [192, 192, 192, 195, 195, 195, 194, 190, 193, 193, 192, 192]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349877673Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"downstairs","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.34989788Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"downstairs","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"Max found gen is 196","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349914228Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"downstairs","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"Generation requested: 196 >= found:196","v":0,"name":"Wait for a query_work_queue command to finish before sending IO crucible","level":30,"time":"2025-11-06T20:19:58.349940035Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"downstairs","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"Next flush: 348","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349956223Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"downstairs","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349972461Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"":"downstairs","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.349989389Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.350004867Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"client":"0","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.350021715Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"client":"1","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.350038353Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"client":"2","":"downstairs","session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.350055441Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.350071299Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"14c6bb53-24af-4208-b8c9-6ab806662b66 is now active with session: bef23214-d5ef-40cc-a3b4-7078b66db151","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.350089226Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.350104524Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990,"session_id":"bef23214-d5ef-40cc-a3b4-7078b66db151"} {"msg":"The guest has finished waiting for activation with:196","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.350120572Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990} {"msg":"Activated sub_volume 14c6bb53-24af-4208-b8c9-6ab806662b66","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:19:58.350134371Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 581 len 12288 data: 12 13 13 02/30 Read block 19 len 32768 03/30 Flush 04/30 Write block 453 len 8192 data: 21 23 05/30 Read block 414 len 20480 06/30 Write block 238 len 20480 data: 17 18 17 18 19 07/30 Read block 338 len 4096 08/30 Read block 38 len 4096 09/30 Flush 10/30 Write block 557 len 32768 data: 17 14 12 11 10 10 10 13 11/30 Write block 245 len 12288 data: 23 20 16 12/30 Read block 186 len 12288 13/30 Read block 241 len 20480 14/30 Read block 43 len 20480 15/30 Read block 502 len 16384 16/30 Flush 17/30 Write block 322 len 36864 data: 9 10 12 13 14 14 14 12 12 18/30 Read block 223 len 36864 19/30 Write block 288 len 32768 data: 13 22 23 20 18 20 20 19 20/30 Write block 304 len 28672 data: 13 15 19 20 19 18 16 21/30 Write block 185 len 32768 data: 18 17 16 16 15 11 13 14 22/30 Write block 186 len 12288 data: 18 17 17 23/30 Read block 343 len 16384 24/30 Write block 252 len 32768 data: 12 12 15 14 13 13 15 8 25/30 Flush 26/30 Write block 302 len 24576 data: 16 13 14 16 20 21 27/30 Write block 376 len 32768 data: 9 12 12 13 15 14 12 13 28/30 Write block 319 len 28672 data: 14 13 14 10 11 13 14 29/30 Read block 162 len 12288 30/30 Read block 79 len 16384 ---------------------------------------------------------------- Crucible gen:196 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Sent Done Done false 1001 Acked Read 8 Sent Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Read 5 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Read 1 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Read 4 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Read 4 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Write 8 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Read 4 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 0 31 Done 0 29 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:19:58.368989095Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1990} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 339 339 339 F F F 1 020-039 192 192 192 339 339 339 F F F 2 040-059 192 192 192 341 341 341 F F F 3 060-079 195 195 195 347 347 347 F F F 4 080-099 195 195 195 347 347 347 F F F 5 100-119 195 195 195 347 347 347 F F F 6 120-139 194 194 194 346 346 346 F F F 7 140-159 190 190 190 336 336 336 F F F 8 160-179 193 193 193 342 342 342 F F F 9 180-199 193 196 196 342 351 351 F F F <--- 10 200-219 192 192 192 339 339 339 F F F 11 220-239 192 196 196 339 349 349 F F F <--- 12 240-259 194 196 196 346 351 351 F F F <--- 13 260-279 194 194 194 346 346 346 F F F 14 280-299 191 196 196 337 351 351 F F F <--- 15 300-319 195 196 196 347 351 351 F T T <--- 16 320-339 192 196 196 339 351 351 F T T <--- 17 340-359 193 193 193 342 342 342 F F F 18 360-379 184 184 184 329 329 329 F T T <--- 19 380-399 195 195 195 347 347 347 F T T <--- 20 400-419 195 195 195 347 347 347 F F F 21 420-439 194 194 194 346 346 346 F F F 22 440-459 192 196 196 340 349 349 F F F <--- 23 460-479 192 192 192 338 338 338 F F F 24 480-499 194 194 194 346 346 346 F F F 25 500-519 194 194 194 346 346 346 F F F 26 520-539 194 194 194 346 346 346 F F F 27 540-559 195 196 196 347 350 350 F F F <--- 28 560-579 189 196 196 335 350 350 F F F <--- 29 580-599 194 196 196 346 348 348 F F F <--- Max gen: 196, Max flush: 351 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 98 Verifying data now No change after verify Loop: 98 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 339 339 339 F F F 1 020-039 192 192 192 339 339 339 F F F 2 040-059 192 192 192 341 341 341 F F F 3 060-079 195 195 195 347 347 347 F F F 4 080-099 195 195 195 347 347 347 F F F 5 100-119 195 195 195 347 347 347 F F F 6 120-139 194 194 194 346 346 346 F F F 7 140-159 190 190 190 336 336 336 F F F 8 160-179 193 193 193 342 342 342 F F F 9 180-199 196 196 196 351 351 351 F F F 10 200-219 192 192 192 339 339 339 F F F 11 220-239 196 196 196 349 349 349 F F F 12 240-259 196 196 196 351 351 351 F F F 13 260-279 194 194 194 346 346 346 F F F 14 280-299 196 196 196 351 351 351 F F F 15 300-319 197 197 197 352 352 352 F F F 16 320-339 197 197 197 352 352 352 F F F 17 340-359 193 193 193 342 342 342 F F F 18 360-379 197 197 197 352 352 352 F F F 19 380-399 197 197 197 352 352 352 F F F 20 400-419 195 195 195 347 347 347 F F F 21 420-439 194 194 194 346 346 346 F F F 22 440-459 196 196 196 349 349 349 F F F 23 460-479 192 192 192 338 338 338 F F F 24 480-499 194 194 194 346 346 346 F F F 25 500-519 194 194 194 346 346 346 F F F 26 520-539 194 194 194 346 346 346 F F F 27 540-559 196 196 196 350 350 350 F F F 28 560-579 196 196 196 350 350 350 F F F 29 580-599 196 196 196 348 348 348 F F F Max gen: 197, Max flush: 352 Begin loop 99 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.798378394Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.843800952Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.844416734Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"51794d4c724f3f3d23d04ef362f9271703c930a0\",\n git_commit_timestamp: \"2025-11-06T19:49:48.000000000Z\",\n git_branch: \"mkeeter/single-block-validation\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.844465728Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.844482026Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.844504113Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998} {"msg":"Crucible 257d4c59-4ba8-46aa-979b-32f8b7c43099 has session id: 2aaf05ec-bda7-490a-86cf-4054c3b49078","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845079361Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"Upstairs opts: Upstairs UUID: 257d4c59-4ba8-46aa-979b-32f8b7c43099, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845141213Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"Crucible stats registered with UUID: 257d4c59-4ba8-46aa-979b-32f8b7c43099","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845175858Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:20:02.845199006Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845367494Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"io task","client":"0","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845416668Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"io task","client":"1","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845446204Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"io task","client":"2","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845596985Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"task":"crutest"} {"msg":"The guest has requested activation with gen:198","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845628212Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998} {"msg":"257d4c59-4ba8-46aa-979b-32f8b7c43099 active request set","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845694823Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.84571511Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"client":"0","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845741627Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"client":"1","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845771563Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"client":"2","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.84580285Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"io task","client":"2","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845834875Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"io task","client":"0","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845863382Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"io task","client":"1","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"ds_connection connected from Ok(127.0.0.1:49706)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.845934833Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"io task","client":"0","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"ds_connection connected from Ok(127.0.0.1:40241)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.846187951Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"io task","client":"1","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"ds_connection connected from Ok(127.0.0.1:41250)","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.846221797Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"io task","client":"2","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.847202033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"client":"0","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.847336866Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"client":"1","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.84778251Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"client":"2","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.847948539Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.847983825Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"[0]R flush_numbers[0..12]: [339, 339, 341, 347, 347, 347, 346, 336, 342, 351, 339, 349]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848326981Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"downstairs","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"[0]R generation[0..12]: [192, 192, 192, 195, 195, 195, 194, 190, 193, 196, 192, 196]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848357728Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"downstairs","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848378895Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"downstairs","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"[1]R flush_numbers[0..12]: [339, 339, 341, 347, 347, 347, 346, 336, 342, 351, 339, 349]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848398152Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"downstairs","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"[1]R generation[0..12]: [192, 192, 192, 195, 195, 195, 194, 190, 193, 196, 192, 196]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.84841705Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"downstairs","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848437038Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"downstairs","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"[2]R flush_numbers[0..12]: [339, 339, 341, 347, 347, 347, 346, 336, 342, 351, 339, 349]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848456035Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"downstairs","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"[2]R generation[0..12]: [192, 192, 192, 195, 195, 195, 194, 190, 193, 196, 192, 196]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848475003Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"downstairs","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.84849473Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"downstairs","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"Max found gen is 198","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848511198Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"downstairs","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"Generation requested: 198 >= found:198","v":0,"Wait for a query_work_queue command to finish before sending IO name":"crucible","level":30,"time":"2025-11-06T20:20:02.848534685Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"downstairs","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"Next flush: 353","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848551003Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"downstairs","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848567251Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"":"downstairs","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848587629Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848603517Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"client":"0","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848620755Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"client":"1","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848637872Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"client":"2","":"downstairs","session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.84865506Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848671258Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msgDisk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ":"257d4c59-4ba8-46aa-979b-32f8b7c43099 is now active with session: 2aaf05ec-bda7-490a-86cf-4054c3b49078","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848692546Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848708033Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998,"session_id":"2aaf05ec-bda7-490a-86cf-4054c3b49078"} {"msg":"The guest has finished waiting for activation with:198","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848725501Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998} {"msg":"Activated sub_volume 257d4c59-4ba8-46aa-979b-32f8b7c43099","v":0,"name":"crucible","level":30,"time":"2025-11-06T20:20:02.848739409Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" Run Repair workload 01/30 Write block 357 len 20480 data: 13 10 9 7 10 02/30 Write block 383 len 12288 data: 14 14 18 03/30 Read block 496 len 12288 04/30 Read block 202 len 8192 05/30 Read block 590 len 36864 06/30 Write block 26 len 40960 data: 18 17 15 12 12 13 14 16 18 21 07/30 Write block 308 len 4096 data: 20 08/30 Read block 408 len 20480 09/30 Read block 226 len 36864 10/30 Read block 42 len 8192 11/30 Flush 12/30 Write block 544 len 12288 data: 21 20 17 13/30 Read block 380 len 24576 14/30 Read block 116 len 28672 15/30 Read block 289 len 8192 16/30 Flush 17/30 Write block 141 len 28672 data: 16 16 15 15 14 11 9 18/30 Flush 19/30 Write block 43 len 40960 data: 18 18 19 19 19 16 14 16 15 17 20/30 Read block 180 len 4096 21/30 Write block 177 len 4096 data: 16 22/30 Write block 273 len 20480 data: 18 18 19 16 18 23/30 Read block 28 len 20480 24/30 Flush 25/30 Write block 129 len 32768 data: 21 19 17 22 23 20 17 16 26/30 Write block 89 len 36864 data: 17 16 17 17 16 15 18 16 16 27/30 Read block 509 len 8192 28/30 Write block 322 len 24576 data: 11 12 14 15 15 15 29/30 Read block 130 len 40960 30/30 Write block 585 len 20480 data: 18 17 18 21 23 ---------------------------------------------------------------- Crucible gen:198 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 3 Done Done Sent false 1002 Acked Read 3 Done Done Sent false 1003 Acked Read 2 Done Done Sent false 1004 Acked Read 9 Done Done Sent false 1005 Acked Write 10 Done Done Sent false 1006 Acked Write 1 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Read 9 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 3 Done Done Sent false 1012 Acked Read 6 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Read 2 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 10 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Write 1 Done Done Sent false 1021 Acked Write 5 Done Done Sent false 1022 Acked Read 5 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 8 Done Done Sent false 1025 Acked Write 9 Done Done Sent false 1026 Acked Read 2 Done Done Sent false 1027 Acked Write 6 Done Done Sent false 1028 Acked Read 10 Done Done Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-11-06T20:20:02.867559808Z","hostname":"w-01K9DCK5D9EW7JXV5KFJAME758","pid":1998} Wrote out file "/tmp/test_live_repair/test_repair_verify.data" Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 339 339 339 F F F 1 020-039 198 198 192 353 353 339 F F F <--- 2 040-059 198 198 192 356 356 341 F F F <--- 3 060-079 195 195 195 347 347 347 F F F 4 080-099 195 195 195 347 347 347 T T F <--- 5 100-119 195 195 195 347 347 347 F F F 6 120-139 194 194 194 346 346 346 T T F <--- 7 140-159 198 198 190 355 355 336 F F F <--- 8 160-179 198 198 193 356 356 342 F F F <--- 9 180-199 196 196 196 351 351 351 F F F 10 200-219 192 192 192 339 339 339 F F F 11 220-239 196 196 196 349 349 349 F F F 12 240-259 196 196 196 351 351 351 F F F 13 260-279 198 198 194 356 356 346 F F F <--- 14 280-299 196 196 196 351 351 351 F F F 15 300-319 198 198 197 353 353 352 F F F <--- 16 320-339 197 197 197 352 352 352 T T F <--- 17 340-359 198 198 193 353 353 342 F F T <--- 18 360-379 198 198 197 353 353 352 F F T <--- 19 380-399 198 198 197 353 353 352 F F F <--- 20 400-419 195 195 195 347 347 347 F F F 21 420-439 194 194 194 346 346 346 F F F 22 440-459 196 196 196 349 349 349 F F F 23 460-479 192 192 192 338 338 338 F F F 24 480-499 194 194 194 346 346 346 F F F 25 500-519 194 194 194 346 346 346 F F F 26 520-539 194 194 194 346 346 346 F F F 27 540-559 198 198 196 354 354 350 F F F <--- 28 560-579 196 196 196 350 350 350 F F F 29 580-599 196 196 196 348 348 348 T T F <--- Max gen: 198, Max flush: 356 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/backtrace.rs:27:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.100/src/lib.rs:695:13 A Difference in extent metadata is expected here On loop 99 Verifying data now No change after verify Loop: 99 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 192 192 192 339 339 339 F F F 1 020-039 198 198 198 353 353 353 F F F 2 040-059 198 198 198 356 356 356 F F F 3 060-079 195 195 195 347 347 347 F F F 4 080-099 199 199 199 357 357 357 F F F 5 100-119 195 195 195 347 347 347 F F F 6 120-139 199 199 199 357 357 357 F F F 7 140-159 198 198 198 355 355 355 F F F 8 160-179 198 198 198 356 356 356 F F F 9 180-199 196 196 196 351 351 351 F F F 10 200-219 192 192 192 339 339 339 F F F 11 220-239 196 196 196 349 349 349 F F F 12 240-259 196 196 196 351 351 351 F F F 13 260-279 198 198 198 356 356 356 F F F 14 280-299 196 196 196 351 351 351 F F F 15 300-319 198 198 198 353 353 353 F F F 16 320-339 199 199 199 357 357 357 F F F 17 340-359 198 198 198 353 353 353 F F F 18 360-379 198 198 198 353 353 353 F F F 19 380-399 198 198 198 353 353 353 F F F 20 400-419 195 195 195 347 347 347 F F F 21 420-439 194 194 194 346 346 346 F F F 22 440-459 196 196 196 349 349 349 F F F 23 460-479 192 192 192 338 338 338 F F F 24 480-499 194 194 194 346 346 346 F F F 25 500-519 194 194 194 346 346 346 F F F 26 520-539 194 194 194 346 346 346 F F F 27 540-559 198 198 198 354 354 354 F F F 28 560-579 196 196 196 350 350 350 F F F 29 580-599 199 199 199 357 357 357 F F F Max gen: 199, Max flush: 357 9:41 Test duration Test completed real 9:40.473213531 user 58.678397970 sys 24.730069786 trap 0.080632497 tflt 0.094496783 dflt 0.127007399 kflt 0.005167064 lock 14:59:51.279573802 slp 46:20.165399782 lat 6.273850832 stop 0.033132508