Turn off color for downstairs dump
Creating region directory at: /var/tmp/test_repair
Creating dsc directory at: /tmp/dsc
Downstairs region 0 created at /var/tmp/test_repair/8810 in 0.29405203
Downstairs region 1 created at /var/tmp/test_repair/8820 in 0.023660349
Downstairs region 2 created at /var/tmp/test_repair/8830 in 0.023363143
Created 3 regions
/var/tmp/bins/crutest with -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 1184 1185 1186
Dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.094171404Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.167437924Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.169713345Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.169830562Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.16984981Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.169865011Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188}
{"msg":"Crucible 54e07938-e843-4f77-bdd7-607807c1bdfe has session id: fba59546-6d91-42fa-98b3-48db22b14b42","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.170967668Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"Upstairs opts: Upstairs UUID: 54e07938-e843-4f77-bdd7-607807c1bdfe, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.171092491Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"Crucible stats registered with UUID: 54e07938-e843-4f77-bdd7-607807c1bdfe","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.171118794Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:14.171175479Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.172326756Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"task":"crutest"}
{"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.172380583Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188}
{"msg":"54e07938-e843-4f77-bdd7-607807c1bdfe active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.172440646Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.172459194Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"0","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.172503407Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"0","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.172527932Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"1","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.172554035Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"1","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.172613299Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"2","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.17272416Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"2","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.172752723Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"io task","client":"0","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.172776438Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"io task","client":"2","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.172799194Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"io task","client":"1","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"ds_connection connected from Ok(127.0.0.1:38987)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.172822529Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"io task","client":"2","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"ds_connection connected from Ok(127.0.0.1:60438)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.172845125Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"io task","client":"0","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"ds_connection connected from Ok(127.0.0.1:33732)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.17286934Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"io task","client":"1","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.174580071Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"2","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.174789161Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"0","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.174833004Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"1","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.175500191Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"2","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.175539527Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"1","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.175568809Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"0","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176426867Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176451072Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.17661567Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"downstairs","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176634908Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"downstairs","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.17664899Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"downstairs","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176662022Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"downstairs","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176675063Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"downstairs","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176688485Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"downstairs","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176701377Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"downstairs","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176714279Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"downstairs","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176730919Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"downstairs","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176743661Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"downstairs","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176763499Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"downstairs","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176785725Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"downstairs","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176809291Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"downstairs","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176824051Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176835684Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"0","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176848596Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"1","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176867484Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"client":"2","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO
fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176892089Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176914525Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"54e07938-e843-4f77-bdd7-607807c1bdfe is now active with session: fba59546-6d91-42fa-98b3-48db22b14b42","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176937061Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176949883Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:14.176961786Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Fill test
Read and Verify all blocks (0..600 range:false)
Wrote out file "/tmp/test_repair_verify.data"
CLIENT: Tests done. All submitted work has been ACK'd
----------------------------------------------------------------
Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:3
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1004 Acked Read 88 Done Done Done false
1005 Acked Read 256 Done Done Done false
1006 Acked Read 256 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 3 3 3 9
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1003 1003 1003
Downstairs last five completed: 1003 1002 1001 1000
Upstairs last five completed: 1006 1005 1004 1003 1002
CLIENT: Up:0 ds:3 act:3
----------------------------------------------------------------
Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1007 1006 1005 1004 1003
Upstairs last five completed: 1007 1006 1005 1004 1003
CLIENT: Up:0 ds:0 act:3
CLIENT: All crucible jobs finished, exiting program
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:18.267955866Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188}
{"msg":"client stopping due to Err(RecvError(()))","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.268107013Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"io task","client":"0","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"client task is sending Done(QueueClosed)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:18.268154923Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"io task","client":"0","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"client task could not reply to main task; shutting down?","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:18.268189212Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"io task","client":"0","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.268221432Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1188,"":"io task","client":"0","":"downstairs","session_id":"fba59546-6d91-42fa-98b3-48db22b14b42"}
Begin loop 1
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.301856943Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.35027569Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.351790572Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.351846267Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.351863647Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.351881705Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190}
{"msg":"Crucible acf99038-c461-4dee-bf11-339697cdf3af has session id: bd7b19c9-e0f4-46b7-926a-a9d343744747","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.352680791Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"Upstairs opts: Upstairs UUID: acf99038-c461-4dee-bf11-339697cdf3af, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.352731479Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"Crucible stats registered with UUID: acf99038-c461-4dee-bf11-339697cdf3af","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.352758423Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:18.3527796Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.353242983Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"task":"crutest"}
{"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.35327858Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190}
{"msg":"acf99038-c461-4dee-bf11-339697cdf3af active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.353337394Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.353362838Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"0","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.353401344Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"0","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.353443668Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"1","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.353471861Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"1","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.353495976Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"2","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.353801077Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"2","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.353846699Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"io task","client":"2","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.353879319Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"io task","client":"0","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.353918954Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"io task","client":"1","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39583)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.353949035Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"io task","client":"0","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"ds_connection connected from Ok(127.0.0.1:44284)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.3539737Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"io task","client":"1","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49246)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.354012976Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"io task","client":"2","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.354775224Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"2","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.354803836Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"0","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.355335397Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"1","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.355420864Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"0","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.355441141Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"2","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.355904334Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"1","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356076368Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.35610491Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356442032Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"downstairs","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356464578Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"downstairs","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356478849Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"downstairs","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356496298Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"downstairs","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.35650971Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"downstairs","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356523361Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"downstairs","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356536563Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"downstairs","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356549655Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"downstairs","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356563007Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"downstairs","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"Max found gen is 2","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:36:18.356575559Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"downstairs","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356593368Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"downstairs","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.35660589Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"downstairs","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356618562Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"":"downstairs","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356633963Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356648564Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"0","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356664714Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"1","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356680334Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"client":"2","":"downstairs","session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356696224Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356713134Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"acf99038-c461-4dee-bf11-339697cdf3af is now active with session: bd7b19c9-e0f4-46b7-926a-a9d343744747","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356729004Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"Set Active after no reconciliation","v":0,Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356746923Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190,"session_id":"bd7b19c9-e0f4-46b7-926a-a9d343744747"}
{"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:18.356761983Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 573 len 32768 data: 2 2 2 2 2 2 2 2
02/30 Write block 275 len 16384 data: 2 2 2 2
03/30 Write block 323 len 20480 data: 2 2 2 2 2
04/30 Read block 309 len 8192
05/30 Flush
06/30 Write block 51 len 32768 data: 2 2 2 2 2 2 2 2
07/30 Read block 247 len 8192
08/30 Write block 274 len 20480 data: 2 3 3 3 3
09/30 Read block 548 len 8192
10/30 Write block 195 len 36864 data: 2 2 2 2 2 2 2 2 2
11/30 Write block 344 len 8192 data: 2 2
12/30 Flush
13/30 Write block 75 len 4096 data: 2
14/30 Read block 460 len 4096
15/30 Read block 428 len 40960
16/30 Write block 194 len 28672 data: 2 3 3 3 3 3 3
17/30 Read block 384 len 32768
18/30 Write block 463 len 4096 data: 2
19/30 Read block 249 len 24576
20/30 Read block 4 len 20480
21/30 Flush
22/30 Write block 65 len 4096 data: 2
23/30 Write block 277 len 24576 data: 4 4 2 2 2 2
24/30 Read block 303 len 20480
25/30 Write block 352 len 36864 data: 2 2 2 2 2 2 2 2 2
26/30 Read block 28 len 32768
27/30 Read block 219 len 36864
28/30 Write block 297 len 20480 data: 2 2 2 2 2
29/30 Read block 212 len 32768
30/30 Write block 151 len 16384 data: 2 2 2 2
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:9
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1021 Acked Write 1 Done Done Done false
1022 Acked Write 6 Done Done Done false
1023 Acked Read 5 Done Done Done false
1024 Acked Write 9 Done Done Done false
1025 Acked Read 8 Done Done Done false
1026 Acked Read 9 Done Done Done false
1027 Acked Write 5 Done Done Done false
1028 Acked Read 8 Done Done Done false
1029 Acked Write 4 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 8 8 8 24
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1020 1020 1020
Downstairs last five completed: 1020 1019 1018 1017 1016
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:18.377537059Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1190}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 1 1 1 1 1 1 F F F
1 020-039 1 1 1 1 1 1 F F F
2 040-059 2 2 2 3 3 3 F F F
3 060-079 2 2 2 4 4 4 T T T
4 080-099 1 1 1 1 1 1 F F F
5 100-119 1 1 1 1 1 1 F F F
6 120-139 1 1 1 1 1 1 F F F
7 140-159 1 1 1 1 1 1 T T T
8 160-179 1 1 1 1 1 1 F F F
9 180-199 2 2 2 4 4 4 F F F
10 200-219 2 2 2 4 4 4 F F F
11 220-239 1 1 1 1 1 1 F F F
12 240-259 1 1 1 1 1 1 F F F
13 260-279 2 2 2 3 3 3 T T T
14 280-299 1 1 1 1 1 1 T T T
15 300-319 1 1 1 1 1 1 T T T
16 320-339 2 2 2 2 2 2 F F F
17 340-359 2 2 2 3 3 3 T T T
18 360-379 1 1 1 1 1 1 T T T
19 380-399 1 1 1 1 1 1 F F F
20 400-419 1 1 1 1 1 1 F F F
21 420-439 1 1 1 1 1 1 F F F
22 440-459 1 1 1 1 1 1 F F F
23 460-479 2 2 2 4 4 4 F F F
24 480-499 1 1 1 1 1 1 F F F
25 500-519 1 1 1 1 1 1 F F F
26 520-539 1 1 1 1 1 1 F F F
27 540-559 1 1 1 1 1 1 F F F
28 560-579 2 2 2 2 2 2 F F F
29 580-599 2 2 2 2 2 2 F F F
Max gen: 2, Max flush: 4
A Difference in extent metadata is expected here
On loop 1
Verifying data now
No change after verify
Loop: 1 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 1 1 1 1 1 1 F F F
1 020-039 1 1 1 1 1 1 F F F
2 040-059 2 2 2 3 3 3 F F F
3 060-079 3 3 3 5 5 5 F F F
4 080-099 1 1 1 1 1 1 F F F
5 100-119 1 1 1 1 1 1 F F F
6 120-139 1 1 1 1 1 1 F F F
7 140-159 3 3 3 5 5 5 F F F
8 160-179 1 1 1 1 1 1 F F F
9 180-199 2 2 2 4 4 4 F F F
10 200-219 2 2 2 4 4 4 F F F
11 220-239 1 1 1 1 1 1 F F F
12 240-259 1 1 1 1 1 1 F F F
13 260-279 3 3 3 5 5 5 F F F
14 280-299 3 3 3 5 5 5 F F F
15 300-319 3 3 3 5 5 5 F F F
16 320-339 2 2 2 2 2 2 F F F
17 340-359 3 3 3 5 5 5 F F F
18 360-379 3 3 3 5 5 5 F F F
19 380-399 1 1 1 1 1 1 F F F
20 400-419 1 1 1 1 1 1 F F F
21 420-439 1 1 1 1 1 1 F F F
22 440-459 1 1 1 1 1 1 F F F
23 460-479 2 2 2 4 4 4 F F F
24 480-499 1 1 1 1 1 1 F F F
25 500-519 1 1 1 1 1 1 F F F
26 520-539 1 1 1 1 1 1 F F F
27 540-559 1 1 1 1 1 1 F F F
28 560-579 2 2 2 2 2 2 F F F
29 580-599 2 2 2 2 2 2 F F F
Max gen: 3, Max flush: 5
Begin loop 2
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.645877818Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.692662867Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.693655702Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.693699874Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.693718843Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.693733094Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198}
{"msg":"Crucible 62c0e5c4-66cc-40f7-8fc5-73b21e3a80a4 has session id: 92f6f4c5-c946-4365-a1dc-832b931afaed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.694339708Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"Upstairs opts: Upstairs UUID: 62c0e5c4-66cc-40f7-8fc5-73b21e3a80a4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.694383461Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"Crucible stats registered with UUID: 62c0e5c4-66cc-40f7-8fc5-73b21e3a80a4","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.694406567Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:22.694426255Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.694815873Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"task":"crutest"}
{"msg":"The guest has requested activation with gen:4","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.694842347Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198}
{"msg":"62c0e5c4-66cc-40f7-8fc5-73b21e3a80a4 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.694903479Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.694920908Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"0","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.694945383Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"0","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.694959474Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"1","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.694974056Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"1","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.694992174Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"2","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.695011462Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"2","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.695035597Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.695060452Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.695083897Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57871)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.695153305Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"io task","client":"0","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"ds_connection connected from Ok(127.0.0.1:64879)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.695206421Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"io task","client":"2","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"ds_connection connected from Ok(127.0.0.1:42588)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.695245237Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"io task","client":"1","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.695980062Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"1","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.696025185Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"0","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.696062391Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"2","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.696652286Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"1","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.696683217Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"2","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.69670991Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"0","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697210351Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.69734018Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"[0]R flush_numbers[0..12]: [1, 1, 3, 5, 1, 1, 1, 5, 1, 4, 4, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697380815Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"downstairs","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"[0]R generation[0..12]: [1, 1, 2, 3, 1, 1, 1, 3, 1, 2, 2, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697397605Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"downstairs","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697411996Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"downstairs","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"[1]R flush_numbers[0..12]: [1, 1, 3, 5, 1, 1, 1, 5, 1, 4, 4, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697425577Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"downstairs","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"[1]R generation[0..12]: [1, 1, 2, 3, 1, 1, 1, 3, 1, 2, 2, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697438709Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"downstairs","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697452221Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"downstairs","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"[2]R flush_numbers[0..12]: [1, 1, 3, 5, 1, 1, 1, 5, 1, 4, 4, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697465393Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"downstairs","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"[2]R generation[0..12]: [1, 1, 2, 3, 1, 1, 1, 3, 1, 2, 2, 1]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697478565Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"downstairs","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697492116Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"downstairs","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"Max found gen is 4","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697504518Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"downstairs","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"Generation requested: 4 >= found:4","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697517211Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"downstairs","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"Next flush: 6","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697529603Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"downstairs","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697541945Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"":"downstairs","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697557546Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":Wait for a query_work_queue command to finish before sending IO
1198,"session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697588017Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"0","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697613191Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"1","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697639655Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"client":"2","":"downstairs","session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697664499Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697689274Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"62c0e5c4-66cc-40f7-8fc5-73b21e3a80a4 is now active with session: 92f6f4c5-c946-4365-a1dc-832b931afaed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697714638Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
2025-01-27T19:36:22.697739663Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198,"session_id":"92f6f4c5-c946-4365-a1dc-832b931afaed"}
{"msg":"The guest has finished waiting for activation with:4","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:22.697768445Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 559 len 28672 data: 2 2 2 2 2 2 2
02/30 Read block 268 len 36864
03/30 Read block 196 len 28672
04/30 Write block 179 len 12288 data: 2 2 2
05/30 Read block 279 len 32768
06/30 Write block 86 len 20480 data: 2 2 2 2 2
07/30 Write block 383 len 40960 data: 2 2 2 2 2 2 2 2 2 2
08/30 Read block 251 len 36864
09/30 Read block 131 len 24576
10/30 Read block 415 len 16384
11/30 Write block 66 len 32768 data: 2 2 2 2 2 2 2 2
12/30 Read block 508 len 20480
13/30 Write block 140 len 4096 data: 2
14/30 Flush
15/30 Write block 143 len 28672 data: 2 2 2 2 2 2 2
16/30 Read block 335 len 24576
17/30 Read block 353 len 16384
18/30 Write block 522 len 40960 data: 2 2 2 2 2 2 2 2 2 2
19/30 Read block 295 len 40960
20/30 Read block 253 len 28672
21/30 Write block 78 len 36864 data: 2 2 2 2 2 2 2 2 3
22/30 Read block 238 len 16384
23/30 Flush
24/30 Write block 233 len 12288 data: 2 2 2
25/30 Read block 451 len 32768
26/30 Write block 159 len 12288 data: 2 2 2
27/30 Read block 182 len 24576
28/30 Read block 420 len 32768
29/30 Read block 547 len 8192
30/30 Write block 472 len 28672 data: 2 2 2 2 2 2 2
----------------------------------------------------------------
Crucible gen:4 GIO:true work queues: Upstairs:0 downstairs:7
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1023 Acked Write 3 Done Done Done false
1024 Acked Read 8 Done Done Done false
1025 Acked Write 3 Done Done Done false
1026 Acked Read 6 Done Done Done false
1027 Acked Read 8 Done Done Done false
1028 Acked Read 2 Done Done Done false
1029 Acked Write 7 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 6 6 6 18
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1022 1022 1022
Downstairs last five completed: 1022 1021 1020 1019 1018
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:22.719402248Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1198}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 1 1 1 1 1 1 F F F
1 020-039 1 1 1 1 1 1 F F F
2 040-059 2 2 2 3 3 3 F F F
3 060-079 4 4 4 7 7 7 F F F
4 080-099 4 4 4 7 7 7 F F F
5 100-119 1 1 1 1 1 1 F F F
6 120-139 1 1 1 1 1 1 F F F
7 140-159 4 4 4 7 7 7 T T T
8 160-179 4 4 4 6 6 6 T T T
9 180-199 4 4 4 6 6 6 F F F
10 200-219 2 2 2 4 4 4 F F F
11 220-239 1 1 1 1 1 1 T T T
12 240-259 1 1 1 1 1 1 F F F
13 260-279 3 3 3 5 5 5 F F F
14 280-299 3 3 3 5 5 5 F F F
15 300-319 3 3 3 5 5 5 F F F
16 320-339 2 2 2 2 2 2 F F F
17 340-359 3 3 3 5 5 5 F F F
18 360-379 3 3 3 5 5 5 F F F
19 380-399 4 4 4 6 6 6 F F F
20 400-419 1 1 1 1 1 1 F F F
21 420-439 1 1 1 1 1 1 F F F
22 440-459 1 1 1 1 1 1 F F F
23 460-479 2 2 2 4 4 4 T T T
24 480-499 1 1 1 1 1 1 F F F
25 500-519 1 1 1 1 1 1 F F F
26 520-539 4 4 4 7 7 7 F F F
27 540-559 4 4 4 6 6 6 F F F
28 560-579 4 4 4 6 6 6 F F F
29 580-599 2 2 2 2 2 2 F F F
Max gen: 4, Max flush: 7
A Difference in extent metadata is expected here
On loop 2
Verifying data now
No change after verify
Loop: 2 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 1 1 1 1 1 1 F F F
1 020-039 1 1 1 1 1 1 F F F
2 040-059 2 2 2 3 3 3 F F F
3 060-079 4 4 4 7 7 7 F F F
4 080-099 4 4 4 7 7 7 F F F
5 100-119 1 1 1 1 1 1 F F F
6 120-139 1 1 1 1 1 1 F F F
7 140-159 5 5 5 8 8 8 F F F
8 160-179 5 5 5 8 8 8 F F F
9 180-199 4 4 4 6 6 6 F F F
10 200-219 2 2 2 4 4 4 F F F
11 220-239 5 5 5 8 8 8 F F F
12 240-259 1 1 1 1 1 1 F F F
13 260-279 3 3 3 5 5 5 F F F
14 280-299 3 3 3 5 5 5 F F F
15 300-319 3 3 3 5 5 5 F F F
16 320-339 2 2 2 2 2 2 F F F
17 340-359 3 3 3 5 5 5 F F F
18 360-379 3 3 3 5 5 5 F F F
19 380-399 4 4 4 6 6 6 F F F
20 400-419 1 1 1 1 1 1 F F F
21 420-439 1 1 1 1 1 1 F F F
22 440-459 1 1 1 1 1 1 F F F
23 460-479 5 5 5 8 8 8 F F F
24 480-499 1 1 1 1 1 1 F F F
25 500-519 1 1 1 1 1 1 F F F
26 520-539 4 4 4 7 7 7 F F F
27 540-559 4 4 4 6 6 6 F F F
28 560-579 4 4 4 6 6 6 F F F
29 580-599 2 2 2 2 2 2 F F F
Max gen: 5, Max flush: 8
Begin loop 3
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:26.96522537Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.012227424Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.013289606Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.013343413Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.013359273Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.013374214Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206}
{"msg":"Crucible 1fe2f8ca-6181-4d2f-8e10-5dddce43f007 has session id: 9e846489-7407-408a-8e8f-9cc913b75a89","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014158678Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"Upstairs opts: Upstairs UUID: 1fe2f8ca-6181-4d2f-8e10-5dddce43f007, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014206458Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"Crucible stats registered with UUID: 1fe2f8ca-6181-4d2f-8e10-5dddce43f007","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014233022Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:27.01425293Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014687281Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"task":"crutest"}
{"msg":"The guest has requested activation with gen:6","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.0147206Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206}
{"msg":"1fe2f8ca-6181-4d2f-8e10-5dddce43f007 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014783661Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.01480154Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"0","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014830832Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"0","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014845523Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"1","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014859934Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"1","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014872786Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"2","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014891694Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"2","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014915649Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014940364Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014959972Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"ds_connection connected from Ok(127.0.0.1:65117)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.014988594Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"ds_connection connected from Ok(127.0.0.1:40476)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.015022463Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"ds_connection connected from Ok(127.0.0.1:43708)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.01505944Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.01581936Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"1","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.015855037Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"0","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.016038874Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"2","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.016580418Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"1","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.016620843Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"2","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.016651124Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"0","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017215385Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017240869Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"[0]R flush_numbers[0..12]: [1, 1, 3, 7, 7, 1, 1, 8, 8, 6, 4, 8]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017297794Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"downstairs","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"[0]R generation[0..12]: [1, 1, 2, 4, 4, 1, 1, 5, 5, 4, 2, 5]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017317522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"downstairs","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017332053Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"downstairs","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"[1]R flush_numbers[0..12]: [1, 1, 3, 7, 7, 1, 1, 8, 8, 6, 4, 8]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017345445Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"downstairs","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"[1]R generation[0..12]: [1, 1, 2, 4, 4, 1, 1, 5, 5, 4, 2, 5]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017358686Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"downstairs","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017372308Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"downstairs","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"[2]R flush_numbers[0..12]: [1, 1, 3, 7, 7, 1, 1, 8, 8, 6, 4, 8]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.01738552Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"downstairs","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"[2]R generation[0..12]: [1, 1, 2, 4, 4, 1, 1, 5, 5, 4, 2, 5]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017399331Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"downstairs","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017413093Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"downstairs","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"Max found gen is 6","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017429133Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"downstairs","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"Generation requested: 6 >= found:6","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:36:27.017442165Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"downstairs","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"Next flush: 9","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017458145Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"downstairs","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017470797Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"":"downstairs","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017483879Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017495562Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"0","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017508853Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"1","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017521596Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"client":"2","":"downstairs","session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017534557Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.01754729Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"1fe2f8ca-6181-4d2f-8e10-5dddce43f007 is now active with session: 9e846489-7407-408a-8e8f-9cc913b75a89","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017560142Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017571825Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206,"session_id":"9e846489-7407-408a-8e8f-9cc913b75a89"}
{"msg":"The guest has finished waiting for activation with:6","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:27.017583887Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 215 len 4096 data: 2
02/30 Read block 507 len 32768
03/30 Read block 257 len 36864
04/30 Read block 293 len 36864
05/30 Write block 297 len 4096 data: 3
06/30 Read block 171 len 20480
07/30 Read block 527 len 24576
08/30 Flush
09/30 Write block 174 len 8192 data: 2 2
10/30 Read block 364 len 36864
11/30 Write block 28 len 32768 data: 2 2 2 2 2 2 2 2
12/30 Read block 44 len 32768
13/30 Write block 228 len 4096 data: 2
14/30 Read block 328 len 32768
15/30 Write block 195 len 16384 data: 4 4 4 4
16/30 Read block 440 len 4096
17/30 Flush
18/30 Write block 393 len 12288 data: 2 2 2
19/30 Read block 423 len 36864
20/30 Write block 319 len 40960 data: 2 2 2 2 3 3 3 3 3 2
21/30 Read block 545 len 24576
22/30 Read block 568 len 32768
23/30 Read block 41 len 12288
24/30 Read block 575 len 12288
25/30 Flush
26/30 Write block 175 len 40960 data: 3 2 2 2 3 3 3 2 2 2
27/30 Write block 62 len 28672 data: 2 2 2 3 3 3 3
28/30 Read block 505 len 16384
29/30 Read block 503 len 32768
30/30 Read block 398 len 40960
----------------------------------------------------------------
Crucible gen:6 GIO:true work queues: Upstairs:0 downstairs:5
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1025 Acked Write 10 Done Done Done false
1026 Acked Write 7 Done Done Done false
1027 Acked Read 4 Done Done Done false
1028 Acked Read 8 Done Done Done false
1029 Acked Read 10 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 1 2
Done 5 4 4 13
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1024 1024 1024
Downstairs last five completed: 1024 1023 1022 1021 1020
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:27.037532814Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1206}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 1 1 1 1 1 1 F F F
1 020-039 6 6 6 10 10 10 F F F
2 040-059 2 2 2 3 3 3 F F F
3 060-079 4 4 4 7 7 7 T T T
4 080-099 4 4 4 7 7 7 F F F
5 100-119 1 1 1 1 1 1 F F F
6 120-139 1 1 1 1 1 1 F F F
7 140-159 5 5 5 8 8 8 F F F
8 160-179 6 6 6 10 10 10 T T T
9 180-199 6 6 6 10 10 10 T T T
10 200-219 6 6 6 9 9 9 F F F
11 220-239 6 6 6 10 10 10 F F F
12 240-259 1 1 1 1 1 1 F F F
13 260-279 3 3 3 5 5 5 F F F
14 280-299 6 6 6 9 9 9 F F F
15 300-319 6 6 6 11 11 11 F F F
16 320-339 6 6 6 11 11 11 F F F
17 340-359 3 3 3 5 5 5 F F F
18 360-379 3 3 3 5 5 5 F F F
19 380-399 6 6 6 11 11 11 F F F
20 400-419 1 1 1 1 1 1 F F F
21 420-439 1 1 1 1 1 1 F F F
22 440-459 1 1 1 1 1 1 F F F
23 460-479 5 5 5 8 8 8 F F F
24 480-499 1 1 1 1 1 1 F F F
25 500-519 1 1 1 1 1 1 F F F
26 520-539 4 4 4 7 7 7 F F F
27 540-559 4 4 4 6 6 6 F F F
28 560-579 4 4 4 6 6 6 F F F
29 580-599 2 2 2 2 2 2 F F F
Max gen: 6, Max flush: 11
A Difference in extent metadata is expected here
On loop 3
Verifying data now
No change after verify
Loop: 3 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 1 1 1 1 1 1 F F F
1 020-039 6 6 6 10 10 10 F F F
2 040-059 2 2 2 3 3 3 F F F
3 060-079 7 7 7 12 12 12 F F F
4 080-099 4 4 4 7 7 7 F F F
5 100-119 1 1 1 1 1 1 F F F
6 120-139 1 1 1 1 1 1 F F F
7 140-159 5 5 5 8 8 8 F F F
8 160-179 7 7 7 12 12 12 F F F
9 180-199 7 7 7 12 12 12 F F F
10 200-219 6 6 6 9 9 9 F F F
11 220-239 6 6 6 10 10 10 F F F
12 240-259 1 1 1 1 1 1 F F F
13 260-279 3 3 3 5 5 5 F F F
14 280-299 6 6 6 9 9 9 F F F
15 300-319 6 6 6 11 11 11 F F F
16 320-339 6 6 6 11 11 11 F F F
17 340-359 3 3 3 5 5 5 F F F
18 360-379 3 3 3 5 5 5 F F F
19 380-399 6 6 6 11 11 11 F F F
20 400-419 1 1 1 1 1 1 F F F
21 420-439 1 1 1 1 1 1 F F F
22 440-459 1 1 1 1 1 1 F F F
23 460-479 5 5 5 8 8 8 F F F
24 480-499 1 1 1 1 1 1 F F F
25 500-519 1 1 1 1 1 1 F F F
26 520-539 4 4 4 7 7 7 F F F
27 540-559 4 4 4 6 6 6 F F F
28 560-579 4 4 4 6 6 6 F F F
29 580-599 2 2 2 2 2 2 F F F
Max gen: 7, Max flush: 12
Begin loop 4
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.28188101Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.327824789Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.329006567Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.329051539Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.32906622Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.329081861Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214}
{"msg":"Crucible 82441ea5-af81-49c2-9f4e-5988d504d5d1 has session id: 9e9d021e-b56f-407a-a8c4-53cd545bce69","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.32971308Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"Upstairs opts: Upstairs UUID: 82441ea5-af81-49c2-9f4e-5988d504d5d1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.329747888Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"Crucible stats registered with UUID: 82441ea5-af81-49c2-9f4e-5988d504d5d1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.329762949Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:31.329775661Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.330396676Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"task":"crutest"}
{"msg":"The guest has requested activation with gen:8","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.330431385Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214}
{"msg":"82441ea5-af81-49c2-9f4e-5988d504d5d1 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.330488939Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.330517841Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"0","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.330559306Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"0","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.330587148Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"1","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.330612443Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"1","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.330635149Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"2","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.33073029Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"2","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.330765938Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"io task","client":"2","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.33079425Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"io task","client":"0","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.330819335Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"io task","client":"1","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"ds_connection connected from Ok(127.0.0.1:65304)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.330934693Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"io task","client":"0","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"ds_connection connected from Ok(127.0.0.1:55112)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.330970481Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"io task","client":"1","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"ds_connection connected from Ok(127.0.0.1:36683)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.330996605Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"io task","client":"2","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.331510586Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"1","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.331658735Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"0","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.331678143Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"2","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.3320697Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"1","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.332104469Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"0","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.332334516Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"2","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.332627865Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.332719518Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"[0]R flush_numbers[0..12]: [1, 10, 3, 12, 7, 1, 1, 8, 12, 12, 9, 10]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.332892311Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"downstairs","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"[0]R generation[0..12]: [1, 6, 2, 7, 4, 1, 1, 5, 7, 7, 6, 6]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.33291057Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"downstairs","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.33292622Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"downstairs","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"[1]R flush_numbers[0..12]: [1, 10, 3, 12, 7, 1, 1, 8, 12, 12, 9, 10]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.332940091Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"downstairs","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"[1]R generation[0..12]: [1, 6, 2, 7, 4, 1, 1, 5, 7, 7, 6, 6]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.332953893Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"downstairs","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.332967704Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"downstairs","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"[2]R flush_numbers[0..12]: [1, 10, 3, 12, 7, 1, 1, 8, 12, 12, 9, 10]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.332981076Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"downstairs","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"[2]R generation[0..12]: [1, 6, 2, 7, 4, 1, 1, 5, 7, 7, 6, 6]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:36:31.333014835Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"downstairs","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.333081584Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"downstairs","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"Max found gen is 8","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.333104919Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"downstairs","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"Generation requested: 8 >= found:8","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.33311947Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"downstairs","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"Next flush: 13","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.333133192Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"downstairs","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.333146274Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"":"downstairs","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.333159585Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.333171478Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"0","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.33318442Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"1","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.333197402Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"client":"2","":"downstairs","session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.333210294Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.333222257Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"82441ea5-af81-49c2-9f4e-5988d504d5d1 is now active with session: 9e9d021e-b56f-407a-a8c4-53cd545bce69","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.333234669Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.33324971Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214,"session_id":"9e9d021e-b56f-407a-a8c4-53cd545bce69"}
{"msg":"The guest has finished waiting for activation with:8","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:31.333262322Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 572 len 24576 data: 2 3 3 3 3 3
02/30 Read block 499 len 16384
03/30 Read block 559 len 28672
04/30 Read block 138 len 36864
05/30 Write block 496 len 28672 data: 2 2 2 2 2 2 2
06/30 Write block 348 len 36864 data: 2 2 2 2 3 3 3 3 3
07/30 Write block 268 len 32768 data: 2 2 2 2 2 2 3 4
08/30 Read block 367 len 8192
09/30 Read block 326 len 12288
10/30 Read block 171 len 20480
11/30 Read block 150 len 36864
12/30 Write block 541 len 4096 data: 2
13/30 Write block 492 len 40960 data: 2 2 2 2 3 3 3 3 3 3
14/30 Write block 343 len 32768 data: 2 3 3 2 2 3 3 3
15/30 Read block 444 len 12288
16/30 Read block 107 len 36864
17/30 Write block 535 len 16384 data: 2 2 2 2
18/30 Read block 368 len 24576
19/30 Read block 400 len 20480
20/30 Read block 587 len 20480
21/30 Write block 30 len 36864 data: 3 3 3 3 3 3 2 2 2
22/30 Read block 577 len 28672
23/30 Read block 305 len 24576
24/30 Write block 115 len 28672 data: 2 2 2 2 2 2 2
25/30 Flush
26/30 Write block 133 len 28672 data: 2 2 2 2 2 2 2
27/30 Write block 413 len 8192 data: 2 2
28/30 Write block 17 len 24576 data: 2 2 2 2 2 2
29/30 Read block 92 len 28672
30/30 Write block 179 len 20480 data: 4 4 4 3 3
----------------------------------------------------------------
Crucible gen:8 GIO:true work queues: Upstairs:0 downstairs:5
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1025 Acked Write 7 Done Done Done false
1026 Acked Write 2 Done Done Done false
1027 Acked Write 6 Done Done Done false
1028 Acked Read 7 Done Done Done false
1029 Acked Write 5 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 4 4 4 12
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1024 1024 1024
Downstairs last five completed: 1024 1023 1022 1021 1020
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:31.353980642Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1214}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 1 1 1 1 1 1 T T T
1 020-039 8 8 8 13 13 13 T T T
2 040-059 2 2 2 3 3 3 F F F
3 060-079 7 7 7 12 12 12 F F F
4 080-099 4 4 4 7 7 7 F F F
5 100-119 8 8 8 13 13 13 F F F
6 120-139 8 8 8 13 13 13 T T T
7 140-159 5 5 5 8 8 8 F F F
8 160-179 7 7 7 12 12 12 T T T
9 180-199 7 7 7 12 12 12 T T T
10 200-219 6 6 6 9 9 9 F F F
11 220-239 6 6 6 10 10 10 F F F
12 240-259 1 1 1 1 1 1 F F F
13 260-279 8 8 8 13 13 13 F F F
14 280-299 6 6 6 9 9 9 F F F
15 300-319 6 6 6 11 11 11 F F F
16 320-339 6 6 6 11 11 11 F F F
17 340-359 8 8 8 13 13 13 F F F
18 360-379 3 3 3 5 5 5 F F F
19 380-399 6 6 6 11 11 11 F F F
20 400-419 1 1 1 1 1 1 T T T
21 420-439 1 1 1 1 1 1 F F F
22 440-459 1 1 1 1 1 1 F F F
23 460-479 5 5 5 8 8 8 F F F
24 480-499 8 8 8 13 13 13 F F F
25 500-519 8 8 8 13 13 13 F F F
26 520-539 8 8 8 13 13 13 F F F
27 540-559 8 8 8 13 13 13 F F F
28 560-579 8 8 8 13 13 13 F F F
29 580-599 2 2 2 2 2 2 F F F
Max gen: 8, Max flush: 13
A Difference in extent metadata is expected here
On loop 4
Verifying data now
No change after verify
Loop: 4 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 9 9 9 14 14 14 F F F
1 020-039 9 9 9 14 14 14 F F F
2 040-059 2 2 2 3 3 3 F F F
3 060-079 7 7 7 12 12 12 F F F
4 080-099 4 4 4 7 7 7 F F F
5 100-119 8 8 8 13 13 13 F F F
6 120-139 9 9 9 14 14 14 F F F
7 140-159 5 5 5 8 8 8 F F F
8 160-179 9 9 9 14 14 14 F F F
9 180-199 9 9 9 14 14 14 F F F
10 200-219 6 6 6 9 9 9 F F F
11 220-239 6 6 6 10 10 10 F F F
12 240-259 1 1 1 1 1 1 F F F
13 260-279 8 8 8 13 13 13 F F F
14 280-299 6 6 6 9 9 9 F F F
15 300-319 6 6 6 11 11 11 F F F
16 320-339 6 6 6 11 11 11 F F F
17 340-359 8 8 8 13 13 13 F F F
18 360-379 3 3 3 5 5 5 F F F
19 380-399 6 6 6 11 11 11 F F F
20 400-419 9 9 9 14 14 14 F F F
21 420-439 1 1 1 1 1 1 F F F
22 440-459 1 1 1 1 1 1 F F F
23 460-479 5 5 5 8 8 8 F F F
24 480-499 8 8 8 13 13 13 F F F
25 500-519 8 8 8 13 13 13 F F F
26 520-539 8 8 8 13 13 13 F F F
27 540-559 8 8 8 13 13 13 F F F
28 560-579 8 8 8 13 13 13 F F F
29 580-599 2 2 2 2 2 2 F F F
Max gen: 9, Max flush: 14
Begin loop 5
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.61692775Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.66400085Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.665212519Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.665277389Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.665303763Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.665320113Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222}
{"msg":"Crucible 7b50a7f6-1374-4e8e-a10d-0d871f31f4f6 has session id: c5a560a9-d9d4-44b9-9249-6d6f4b81fc73","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.665926068Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"Upstairs opts: Upstairs UUID: 7b50a7f6-1374-4e8e-a10d-0d871f31f4f6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.665978375Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"Crucible stats registered with UUID: 7b50a7f6-1374-4e8e-a10d-0d871f31f4f6","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.66600293Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:35.66601806Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.666456659Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"task":"crutest"}
{"msg":"The guest has requested activation with gen:10","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.666481744Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222}
{"msg":"7b50a7f6-1374-4e8e-a10d-0d871f31f4f6 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.666543345Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.66658502Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"0","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.666702567Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"0","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.666735027Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"1","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.666764628Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"1","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.667038928Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"2","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.66713277Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"2","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.667242892Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"io task","client":"2","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.667272963Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"io task","client":"0","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.667299897Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"io task","client":"1","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"ds_connection connected from Ok(127.0.0.1:45781)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.667329878Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"io task","client":"0","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"ds_connection connected from Ok(127.0.0.1:54740)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.667355522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"io task","client":"2","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"ds_connection connected from Ok(127.0.0.1:61796)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.667380217Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"io task","client":"1","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.667960618Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"2","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.668028346Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"0","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.668239545Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"1","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.668820076Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"0","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.668859691Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"1","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.668935764Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"2","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669436614Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669462897Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"[0]R flush_numbers[0..12]: [14, 14, 3, 12, 7, 13, 14, 8, 14, 14, 9, 10]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.66949023Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"downstairs","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"[0]R generation[0..12]: [9, 9, 2, 7, 4, 8, 9, 5, 9, 9, 6, 6]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669506081Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"downstairs","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669520262Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"downstairs","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"[1]R flush_numbers[0..12]: [14, 14, 3, 12, 7, 13, 14, 8, 14, 14, 9, 10]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.66953887Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"downstairs","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"[1]R generation[0..12]: [9, 9, 2, 7, 4, 8, 9, 5, 9, 9, 6, 6]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669552902Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"downstairs","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669566903Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"downstairs","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"[2]R flush_numbers[0..12]: [14, 14, 3, 12, 7, 13, 14, 8, 14, 14, 9, 10]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669580225Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"downstairs","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"[2]R generation[0..12]: [9, 9, 2, 7, 4, 8, 9, 5, 9, 9, 6, 6]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669593247Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"downstairs","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669606778Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"downstairs","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"Max found gen is 10","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669619421Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"downstairs","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"Generation requested: 10 >= found:10","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669632153Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"downstairs","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"Next flush: 15","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669644605Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"":"downstairs","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669657087Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid"Wait for a query_work_queue command to finish before sending IO
:1222,"":"downstairs","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669686119Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669709245Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"0","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669724595Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"1","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669737697Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"client":"2","":"downstairs","session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669750669Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.66976532Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"7b50a7f6-1374-4e8e-a10d-0d871f31f4f6 is now active with session: c5a560a9-d9d4-44b9-9249-6d6f4b81fc73","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669787117Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669807444Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222,"session_id":"c5a560a9-d9d4-44b9-9249-6d6f4b81fc73"}
{"msg":"The guest has finished waiting for activation with:10","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:35.669820106Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 339 len 16384 data: 2 2 2 2
02/30 Read block 84 len 12288
03/30 Read block 336 len 40960
04/30 Read block 111 len 16384
05/30 Read block 438 len 16384
06/30 Read block 387 len 32768
07/30 Read block 69 len 20480
08/30 Write block 527 len 36864 data: 3 3 3 3 3 2 2 2 3
09/30 Read block 423 len 8192
10/30 Flush
11/30 Write block 63 len 36864 data: 3 3 4 4 4 4 3 3 3
12/30 Write block 502 len 12288 data: 3 2 2
13/30 Write block 288 len 24576 data: 2 2 2 2 2 2
14/30 Read block 367 len 12288
15/30 Write block 7 len 36864 data: 2 2 2 2 2 2 2 2 2
16/30 Read block 293 len 12288
17/30 Read block 52 len 32768
18/30 Write block 200 len 36864 data: 4 3 3 3 2 2 2 2 2
19/30 Write block 454 len 32768 data: 2 2 2 2 2 2 2 2
20/30 Read block 184 len 16384
21/30 Write block 377 len 16384 data: 2 2 2 2
22/30 Write block 392 len 4096 data: 3
23/30 Write block 527 len 32768 data: 4 4 4 4 4 3 3 3
24/30 Flush
25/30 Write block 581 len 24576 data: 2 2 2 2 2 2
26/30 Read block 501 len 16384
27/30 Read block 134 len 24576
28/30 Read block 572 len 16384
29/30 Write block 303 len 32768 data: 2 2 2 2 2 2 2 2
30/30 Read block 239 len 32768
----------------------------------------------------------------
Crucible gen:10 GIO:true work queues: Upstairs:0 downstairs:6
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1024 Acked Write 6 Done Done Done false
1025 Acked Read 4 Done Done Done false
1026 Acked Read 6 Done Done Done false
1027 Acked Read 4 Done Done Done false
1028 Acked Write 8 Done Done Sent false
1029 Acked Read 8 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 2 3
Done 6 5 4 15
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1023 1023 1023
Downstairs last five completed: 1023 1022 1021 1020 1019
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:35.691605516Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1222}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 10 10 10 16 16 16 F F F
1 020-039 9 9 9 14 14 14 F F F
2 040-059 2 2 2 3 3 3 F F F
3 060-079 10 10 10 16 16 16 F F F
4 080-099 4 4 4 7 7 7 F F F
5 100-119 8 8 8 13 13 13 F F F
6 120-139 9 9 9 14 14 14 F F F
7 140-159 5 5 5 8 8 8 F F F
8 160-179 9 9 9 14 14 14 F F F
9 180-199 9 9 9 14 14 14 F F F
10 200-219 10 10 10 16 16 16 F F F
11 220-239 6 6 6 10 10 10 F F F
12 240-259 1 1 1 1 1 1 F F F
13 260-279 8 8 8 13 13 13 F F F
14 280-299 10 10 10 16 16 16 F F F
15 300-319 6 6 6 11 11 11 T T T
16 320-339 10 10 10 15 15 15 F F F
17 340-359 10 10 10 15 15 15 F F F
18 360-379 10 10 10 16 16 16 F F F
19 380-399 10 10 10 16 16 16 F F F
20 400-419 9 9 9 14 14 14 F F F
21 420-439 1 1 1 1 1 1 F F F
22 440-459 10 10 10 16 16 16 F F F
23 460-479 10 10 10 16 16 16 F F F
24 480-499 8 8 8 13 13 13 F F F
25 500-519 10 10 10 16 16 16 F F F
26 520-539 10 10 10 16 16 16 F F F
27 540-559 8 8 8 13 13 13 F F F
28 560-579 8 8 8 13 13 13 F F F
29 580-599 2 2 2 2 2 2 T T T
Max gen: 10, Max flush: 16
A Difference in extent metadata is expected here
On loop 5
Verifying data now
No change after verify
Loop: 5 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 10 10 10 16 16 16 F F F
1 020-039 9 9 9 14 14 14 F F F
2 040-059 2 2 2 3 3 3 F F F
3 060-079 10 10 10 16 16 16 F F F
4 080-099 4 4 4 7 7 7 F F F
5 100-119 8 8 8 13 13 13 F F F
6 120-139 9 9 9 14 14 14 F F F
7 140-159 5 5 5 8 8 8 F F F
8 160-179 9 9 9 14 14 14 F F F
9 180-199 9 9 9 14 14 14 F F F
10 200-219 10 10 10 16 16 16 F F F
11 220-239 6 6 6 10 10 10 F F F
12 240-259 1 1 1 1 1 1 F F F
13 260-279 8 8 8 13 13 13 F F F
14 280-299 10 10 10 16 16 16 F F F
15 300-319 11 11 11 17 17 17 F F F
16 320-339 10 10 10 15 15 15 F F F
17 340-359 10 10 10 15 15 15 F F F
18 360-379 10 10 10 16 16 16 F F F
19 380-399 10 10 10 16 16 16 F F F
20 400-419 9 9 9 14 14 14 F F F
21 420-439 1 1 1 1 1 1 F F F
22 440-459 10 10 10 16 16 16 F F F
23 460-479 10 10 10 16 16 16 F F F
24 480-499 8 8 8 13 13 13 F F F
25 500-519 10 10 10 16 16 16 F F F
26 520-539 10 10 10 16 16 16 F F F
27 540-559 8 8 8 13 13 13 F F F
28 560-579 8 8 8 13 13 13 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 11, Max flush: 17
Begin loop 6
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.925465186Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.971279046Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.972320561Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.972364304Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.972385031Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.972399822Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230}
{"msg":"Crucible ed1d5534-b4bd-4000-a9da-d2ec3a9efd6e has session id: 157d9048-e325-413e-8d65-64d46e6893e4","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.9729529Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"Upstairs opts: Upstairs UUID: ed1d5534-b4bd-4000-a9da-d2ec3a9efd6e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.973001079Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"Crucible stats registered with UUID: ed1d5534-b4bd-4000-a9da-d2ec3a9efd6e","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.973020018Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:39.973088595Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.973707822Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"task":"crutest"}
{"msg":"The guest has requested activation with gen:12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.973736184Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230}
{"msg":"ed1d5534-b4bd-4000-a9da-d2ec3a9efd6e active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.973812237Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.973831265Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"0","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.973857999Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"0","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.97387232Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"1","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.973902241Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"1","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.973931483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"2","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.973967401Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"2","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.973997902Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"io task","client":"2","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.974025225Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"io task","client":"0","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.974052289Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"io task","client":"1","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52338)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.974088066Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"io task","client":"2","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"ds_connection connected from Ok(127.0.0.1:55387)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.974126063Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"io task","client":"0","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"ds_connection connected from Ok(127.0.0.1:45325)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.974262229Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"io task","client":"1","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.975079523Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"0","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.975108445Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"1","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.975673135Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"2","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.975854403Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"1","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.97587452Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"0","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.976303574Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"2","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.976503981Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.976592206Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"[0]R flush_numbers[0..12]: [16, 14, 3, 16, 7, 13, 14, 8, 14, 14, 16, 10]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.976858521Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"downstairs","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"[0]R generation[0..12]: [10, 9, 2, 10, 4, 8, 9, 5, 9, 9, 10, 6]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.976881647Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"downstairs","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.976896457Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"downstairs","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"[1]R flush_numbers[0..12]: [16, 14, 3, 16, 7, 13, 14, 8, 14, 14, 16, 10]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.976909919Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"downstairs","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"[1]R generation[0..12]: [10, 9, 2, 10, 4, 8, 9, 5, 9, 9, 10, 6]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.976923131Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"downstairs","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.976936533Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"downstairs","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"[2]R flush_numbers[0..12]: [16, 14, 3, 16, 7, 13, 14, 8, 14, 14, 16, 10]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.976950424Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"downstairs","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"[2]R generation[0..12]: [10, 9, 2, 10, 4, 8, 9, 5, 9, 9, 10, 6]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.976963576Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"downstairs","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.976976977Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"downstairs","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"Max found gen is 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.97698948Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"downstairs","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"Generation requested: 12 >= found:12","v":0,"name":"crucible","level":30,"Wait for a query_work_queue command to finish before sending IO
time":"2025-01-27T19:36:39.977002462Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"downstairs","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"Next flush: 18","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.977023269Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"downstairs","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.977035681Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"":"downstairs","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.977048513Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.977063504Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"0","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.977076686Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"1","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.977089398Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"client":"2","":"downstairs","session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.97710204Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.977113973Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"ed1d5534-b4bd-4000-a9da-d2ec3a9efd6e is now active with session: 157d9048-e325-413e-8d65-64d46e6893e4","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.977126535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.977138178Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230,"session_id":"157d9048-e325-413e-8d65-64d46e6893e4"}
{"msg":"The guest has finished waiting for activation with:12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:39.97714993Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 254 len 8192 data: 2 2
02/30 Read block 76 len 28672
03/30 Write block 109 len 16384 data: 2 2 2 2
04/30 Write block 144 len 20480 data: 3 3 3 3 3
05/30 Write block 204 len 36864 data: 3 3 3 3 3 2 2 2 2
06/30 Write block 300 len 8192 data: 3 3
07/30 Write block 205 len 4096 data: 4
08/30 Write block 146 len 16384 data: 4 4 4 3
09/30 Write block 406 len 36864 data: 2 2 2 2 2 2 2 3 3
10/30 Write block 524 len 8192 data: 3 3
11/30 Flush
12/30 Write block 551 len 40960 data: 2 2 2 2 2 2 2 2 3 3
13/30 Read block 504 len 4096
14/30 Write block 322 len 16384 data: 3 4 4 4
15/30 Read block 60 len 16384
16/30 Write block 299 len 24576 data: 3 4 4 2 3 3
17/30 Flush
18/30 Write block 373 len 32768 data: 2 2 2 2 3 3 3 3
19/30 Read block 400 len 28672
20/30 Write block 423 len 12288 data: 2 2 2
21/30 Write block 528 len 40960 data: 5 5 5 5 4 4 4 4 3 3
22/30 Read block 206 len 20480
23/30 Write block 102 len 4096 data: 2
24/30 Write block 386 len 20480 data: 3 3 3 3 3
25/30 Write block 521 len 16384 data: 2 3 3 4
26/30 Read block 224 len 8192
27/30 Write block 153 len 4096 data: 3
28/30 Write block 145 len 24576 data: 4 5 5 5 4 2
29/30 Write block 27 len 28672 data: 2 3 3 4 4 4 4
30/30 Read block 88 len 40960
----------------------------------------------------------------
Crucible gen:12 GIO:true work queues: Upstairs:0 downstairs:13
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1017 Acked Write 8 Done Done Done false
1018 Acked Read 7 Done Done Done false
1019 Acked Write 3 Done Done Done false
1020 Acked Write 10 Done Done Done false
1021 Acked Read 5 Done Done Done false
1022 Acked Write 1 Done Done Done false
1023 Acked Write 5 Done Done Done false
1024 Acked Write 4 Done Done Done false
1025 Acked Read 2 Done Done Done false
1026 Acked Write 1 Done Done Done false
1027 Acked Write 6 Done Done Done false
1028 Acked Write 7 Done Done Done false
1029 Acked Read 10 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 1 1
Done 13 13 12 38
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1016 1016 1016
Downstairs last five completed: 1016 1015 1014 1013 1012
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:39.994990173Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1230}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 10 10 10 16 16 16 F F F
1 020-039 9 9 9 14 14 14 T T T
2 040-059 2 2 2 3 3 3 F F F
3 060-079 10 10 10 16 16 16 F F F
4 080-099 4 4 4 7 7 7 F F F
5 100-119 12 12 12 18 18 18 T T T
6 120-139 9 9 9 14 14 14 F F F
7 140-159 12 12 12 18 18 18 T T T
8 160-179 9 9 9 14 14 14 F F F
9 180-199 9 9 9 14 14 14 F F F
10 200-219 12 12 12 18 18 18 F F F
11 220-239 6 6 6 10 10 10 F F F
12 240-259 12 12 12 18 18 18 F F F
13 260-279 8 8 8 13 13 13 F F F
14 280-299 12 12 12 19 19 19 F F F
15 300-319 12 12 12 19 19 19 F F F
16 320-339 12 12 12 19 19 19 F F F
17 340-359 10 10 10 15 15 15 F F F
18 360-379 10 10 10 16 16 16 T T T
19 380-399 10 10 10 16 16 16 T T T
20 400-419 12 12 12 18 18 18 F F F
21 420-439 1 1 1 1 1 1 T T T
22 440-459 10 10 10 16 16 16 F F F
23 460-479 10 10 10 16 16 16 F F F
24 480-499 8 8 8 13 13 13 F F F
25 500-519 10 10 10 16 16 16 F F F
26 520-539 12 12 12 18 18 18 T T T
27 540-559 12 12 12 19 19 19 F F F
28 560-579 12 12 12 19 19 19 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 12, Max flush: 19
A Difference in extent metadata is expected here
On loop 6
Verifying data now
No change after verify
Loop: 6 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 10 10 10 16 16 16 F F F
1 020-039 13 13 13 20 20 20 F F F
2 040-059 2 2 2 3 3 3 F F F
3 060-079 10 10 10 16 16 16 F F F
4 080-099 4 4 4 7 7 7 F F F
5 100-119 13 13 13 20 20 20 F F F
6 120-139 9 9 9 14 14 14 F F F
7 140-159 13 13 13 20 20 20 F F F
8 160-179 9 9 9 14 14 14 F F F
9 180-199 9 9 9 14 14 14 F F F
10 200-219 12 12 12 18 18 18 F F F
11 220-239 6 6 6 10 10 10 F F F
12 240-259 12 12 12 18 18 18 F F F
13 260-279 8 8 8 13 13 13 F F F
14 280-299 12 12 12 19 19 19 F F F
15 300-319 12 12 12 19 19 19 F F F
16 320-339 12 12 12 19 19 19 F F F
17 340-359 10 10 10 15 15 15 F F F
18 360-379 13 13 13 20 20 20 F F F
19 380-399 13 13 13 20 20 20 F F F
20 400-419 12 12 12 18 18 18 F F F
21 420-439 13 13 13 20 20 20 F F F
22 440-459 10 10 10 16 16 16 F F F
23 460-479 10 10 10 16 16 16 F F F
24 480-499 8 8 8 13 13 13 F F F
25 500-519 10 10 10 16 16 16 F F F
26 520-539 13 13 13 20 20 20 F F F
27 540-559 12 12 12 19 19 19 F F F
28 560-579 12 12 12 19 19 19 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 13, Max flush: 20
Begin loop 7
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.26163813Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.306619216Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.307452899Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.307498921Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.307513911Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.307528693Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238}
{"msg":"Crucible d1a43ef9-7e97-4659-a3e1-fe96b1cfee8a has session id: e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308149748Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"Upstairs opts: Upstairs UUID: d1a43ef9-7e97-4659-a3e1-fe96b1cfee8a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308190043Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"Crucible stats registered with UUID: d1a43ef9-7e97-4659-a3e1-fe96b1cfee8a","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308205773Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:44.308218035Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308652986Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"task":"crutest"}
{"msg":"The guest has requested activation with gen:14","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308707672Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238}
{"msg":"d1a43ef9-7e97-4659-a3e1-fe96b1cfee8a active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308742121Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308756482Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"0","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308778638Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"0","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.30879206Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"1","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308806071Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"1","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308825799Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"2","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308851483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"2","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308878517Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"io task","client":"2","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308909278Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"io task","client":"1","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308936241Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"io task","client":"0","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"ds_connection connected from Ok(127.0.0.1:33603)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308961435Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"io task","client":"2","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39818)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.308989228Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"io task","client":"0","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"ds_connection connected from Ok(127.0.0.1:40514)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.309387711Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"io task","client":"1","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.309876639Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"0","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.310059055Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"1","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.3104384Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"0","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.310575886Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"1","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.310664241Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"2","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.31114967Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.311265279Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.311524268Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"2","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"[0]R flush_numbers[0..12]: [16, 20, 3, 16, 7, 20, 14, 20, 14, 14, 18, 10]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312101881Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"downstairs","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"[0]R generation[0..12]: [10, 13, 2, 10, 4, 13, 9, 13, 9, 9, 12, 6]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312133401Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"downstairs","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312149141Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"downstairs","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"[1]R flush_numbers[0..12]: [16, 20, 3, 16, 7, 20, 14, 20, 14, 14, 18, 10]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312163152Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"downstairs","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"[1]R generation[0..12]: [10, 13, 2, 10, 4, 13, 9, 13, 9, 9, 12, 6]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312176544Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"downstairs","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312190356Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"downstairs","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"[2]R flush_numbers[0..12]: [16, 20, 3, 16, 7, 20, 14, 20, 14, 14, 18, 10]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312203567Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"downstairs","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"[2]R generation[0..12]: [10, 13, 2, 10, 4, 13, 9, 13, 9, 9, 12, 6]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.31221657Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"downstairs","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312235038Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"downstairs","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"Max found gen is 14","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.31224791Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"downstairs","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"Generation requested: 14 >= found:14","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312260712Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"downstairs","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"Next flush: 21","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312273124Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"downstairs","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"All extents match","v":0,"name":"Wait for a query_work_queue command to finish before sending IO
crucible","level":30,"time":"2025-01-27T19:36:44.312292552Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"":"downstairs","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312305944Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312317667Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"0","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312330529Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"1","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312343011Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"client":"2","":"downstairs","session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312355803Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312367586Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"d1a43ef9-7e97-4659-a3e1-fe96b1cfee8a is now active with session: e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312380018Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312391511Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238,"session_id":"e06a1b79-b5a9-4f6c-9d3d-2e42abcd7400"}
{"msg":"The guest has finished waiting for activation with:14","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:44.312403484Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 439 len 12288 data: 2 2 2
02/30 Read block 430 len 16384
03/30 Read block 122 len 12288
04/30 Read block 143 len 32768
05/30 Write block 181 len 28672 data: 5 4 4 3 2 2 2
06/30 Read block 337 len 12288
07/30 Write block 369 len 16384 data: 2 2 2 2
08/30 Read block 249 len 20480
09/30 Write block 559 len 4096 data: 4
10/30 Write block 171 len 32768 data: 2 2 2 3 4 3 3 3
11/30 Write block 527 len 36864 data: 5 6 6 6 6 5 5 5 5
12/30 Write block 400 len 40960 data: 2 2 2 2 2 2 3 3 3 3
13/30 Write block 506 len 28672 data: 2 2 2 2 2 2 2
14/30 Write block 313 len 8192 data: 2 2
15/30 Read block 95 len 40960
16/30 Flush
17/30 Flush
18/30 Write block 153 len 8192 data: 4 3
19/30 Read block 198 len 40960
20/30 Write block 274 len 36864 data: 4 5 4 5 5 3 3 3 3
21/30 Write block 356 len 24576 data: 4 3 3 3 3 2
22/30 Read block 280 len 16384
23/30 Read block 144 len 36864
24/30 Read block 487 len 12288
25/30 Write block 230 len 8192 data: 2 2
26/30 Flush
27/30 Write block 346 len 16384 data: 3 3 4 4
28/30 Write block 417 len 28672 data: 2 2 2 2 2 2 3
29/30 Read block 248 len 8192
30/30 Read block 354 len 12288
----------------------------------------------------------------
Crucible gen:14 GIO:true work queues: Upstairs:0 downstairs:4
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1026 Acked Write 4 Done Done Done false
1027 Acked Write 7 Done Done Done false
1028 Acked Read 2 Done Done Done false
1029 Acked Read 3 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 0 1
Done 4 3 4 11
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1025 1025 1025
Downstairs last five completed: 1025 1024 1023 1022 1021
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:44.333512632Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1238}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 10 10 10 16 16 16 F F F
1 020-039 13 13 13 20 20 20 F F F
2 040-059 2 2 2 3 3 3 F F F
3 060-079 10 10 10 16 16 16 F F F
4 080-099 4 4 4 7 7 7 F F F
5 100-119 13 13 13 20 20 20 F F F
6 120-139 9 9 9 14 14 14 F F F
7 140-159 14 14 14 23 23 23 F F F
8 160-179 14 14 14 21 21 21 F F F
9 180-199 14 14 14 21 21 21 F F F
10 200-219 12 12 12 18 18 18 F F F
11 220-239 14 14 14 23 23 23 F F F
12 240-259 12 12 12 18 18 18 F F F
13 260-279 14 14 14 23 23 23 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 14 14 14 21 21 21 F F F
16 320-339 12 12 12 19 19 19 F F F
17 340-359 14 14 14 23 23 23 T T T
18 360-379 14 14 14 23 23 23 F F F
19 380-399 13 13 13 20 20 20 F F F
20 400-419 14 14 14 21 21 21 T T T
21 420-439 14 14 14 21 21 21 T T T
22 440-459 14 14 14 21 21 21 F F F
23 460-479 10 10 10 16 16 16 F F F
24 480-499 8 8 8 13 13 13 F F F
25 500-519 14 14 14 21 21 21 F F F
26 520-539 14 14 14 21 21 21 F F F
27 540-559 14 14 14 21 21 21 F F F
28 560-579 12 12 12 19 19 19 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 14, Max flush: 23
A Difference in extent metadata is expected here
On loop 7
Verifying data now
No change after verify
Loop: 7 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 10 10 10 16 16 16 F F F
1 020-039 13 13 13 20 20 20 F F F
2 040-059 2 2 2 3 3 3 F F F
3 060-079 10 10 10 16 16 16 F F F
4 080-099 4 4 4 7 7 7 F F F
5 100-119 13 13 13 20 20 20 F F F
6 120-139 9 9 9 14 14 14 F F F
7 140-159 14 14 14 23 23 23 F F F
8 160-179 14 14 14 21 21 21 F F F
9 180-199 14 14 14 21 21 21 F F F
10 200-219 12 12 12 18 18 18 F F F
11 220-239 14 14 14 23 23 23 F F F
12 240-259 12 12 12 18 18 18 F F F
13 260-279 14 14 14 23 23 23 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 14 14 14 21 21 21 F F F
16 320-339 12 12 12 19 19 19 F F F
17 340-359 15 15 15 24 24 24 F F F
18 360-379 14 14 14 23 23 23 F F F
19 380-399 13 13 13 20 20 20 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 15 15 15 24 24 24 F F F
22 440-459 14 14 14 21 21 21 F F F
23 460-479 10 10 10 16 16 16 F F F
24 480-499 8 8 8 13 13 13 F F F
25 500-519 14 14 14 21 21 21 F F F
26 520-539 14 14 14 21 21 21 F F F
27 540-559 14 14 14 21 21 21 F F F
28 560-579 12 12 12 19 19 19 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 15, Max flush: 24
Begin loop 8
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.585115425Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.632349935Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.633422871Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.633476448Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.633495376Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.633513045Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246}
{"msg":"Crucible adc1d9b3-68aa-49f2-852b-b48f3c8151cf has session id: ea73c398-1fcf-484d-b766-1090c47c81dd","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634161284Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"Upstairs opts: Upstairs UUID: adc1d9b3-68aa-49f2-852b-b48f3c8151cf, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634205616Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"Crucible stats registered with UUID: adc1d9b3-68aa-49f2-852b-b48f3c8151cf","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634226553Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:48.634239465Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634673866Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"task":"crutest"}
{"msg":"The guest has requested activation with gen:16","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634703797Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246}
{"msg":"adc1d9b3-68aa-49f2-852b-b48f3c8151cf active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634769847Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634787436Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"0","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634810701Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"0","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634824253Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"1","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634838074Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"1","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634853984Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"2","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634879279Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"2","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634904204Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"io task","client":"2","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634929088Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"io task","client":"0","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634954672Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"io task","client":"1","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:62267)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.634978168Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"io task","client":"2","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:60892)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.635008569Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"io task","client":"0","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:51498)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.635298989Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"io task","client":"1","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.635792853Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"0","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.635831589Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"1","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.635984944Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"2","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.636516805Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"1","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.636545327Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"0","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.636639319Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"2","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637128196Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637238488Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"[0]R flush_numbers[0..12]: [16, 20, 3, 16, 7, 20, 14, 23, 21, 21, 18, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637280762Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"downstairs","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"[0]R generation[0..12]: [10, 13, 2, 10, 4, 13, 9, 14, 14, 14, 12, 14]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637297781Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"downstairs","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637312212Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"downstairs","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"[1]R flush_numbers[0..12]: [16, 20, 3, 16, 7, 20, 14, 23, 21, 21, 18, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637325664Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"downstairs","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"[1]R generation[0..12]: [10, 13, 2, 10, 4, 13, 9, 14, 14, 14, 12, 14]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637338946Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"downstairs","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637356515Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"downstairs","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"[2]R flush_numbers[0..12]: [16, 20, 3, 16, 7, 20, 14, 23, 21, 21, 18, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637370626Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"downstairs","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"[2]R generation[0..12]: [10, 13, 2, 10, 4, 13, 9, 14, 14, 14, 12, 14]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637383758Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"downstairs","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.63739723Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"downstairs","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"Max found gen is 16","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:36:48.637415948Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"downstairs","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"Generation requested: 16 >= found:16","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637433937Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"downstairs","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"Next flush: 25","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637446469Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"downstairs","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637458951Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"":"downstairs","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637471863Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637483826Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"0","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637496798Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"1","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.63750941Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"client":"2","":"downstairs","session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637522312Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637534155Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"adc1d9b3-68aa-49f2-852b-b48f3c8151cf is now active with session: ea73c398-1fcf-484d-b766-1090c47c81dd","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637546597Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637559939Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246,"session_id":"ea73c398-1fcf-484d-b766-1090c47c81dd"}
{"msg":"The guest has finished waiting for activation with:16","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:48.637571852Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 269 len 40960 data: 3 3 3 3 3 5 6 5 6 6
02/30 Write block 251 len 28672 data: 2 2 2 3 3 2 2
03/30 Write block 78 len 4096 data: 3
04/30 Write block 434 len 20480 data: 2 2 2 2 2
05/30 Flush
06/30 Write block 26 len 20480 data: 2 3 4 4 5
07/30 Read block 286 len 36864
08/30 Write block 70 len 36864 data: 4 4 3 3 2 3 2 2 4
09/30 Read block 72 len 20480
10/30 Read block 537 len 40960
11/30 Read block 110 len 8192
12/30 Flush
13/30 Write block 492 len 8192 data: 3 3
14/30 Read block 174 len 20480
15/30 Write block 40 len 32768 data: 2 2 2 2 2 2 2 2
16/30 Read block 193 len 36864
17/30 Write block 565 len 32768 data: 3 2 2 2 2 2 2 3
18/30 Write block 207 len 32768 data: 4 4 3 3 3 3 2 2
19/30 Read block 454 len 40960
20/30 Flush
21/30 Write block 176 len 20480 data: 4 4 4 5 5
22/30 Write block 90 len 28672 data: 3 2 2 2 2 2 2
23/30 Write block 116 len 16384 data: 3 3 3 3
24/30 Write block 520 len 40960 data: 2 3 4 4 5 4 3 6 7 7
25/30 Write block 480 len 36864 data: 2 2 2 2 2 2 2 2 2
26/30 Read block 251 len 32768
27/30 Read block 558 len 12288
28/30 Write block 65 len 4096 data: 5
29/30 Write block 303 len 40960 data: 4 4 3 3 3 3 3 3 2 2
30/30 Read block 457 len 24576
----------------------------------------------------------------
Crucible gen:16 GIO:true work queues: Upstairs:0 downstairs:10
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1020 Acked Write 5 Done Done Done false
1021 Acked Write 7 Done Done Done false
1022 Acked Write 4 Done Done Done false
1023 Acked Write 10 Done Done Done false
1024 Acked Write 9 Done Done Done false
1025 Acked Read 8 Done Done Done false
1026 Acked Read 3 Done Done Done false
1027 Acked Write 1 Done Done Done false
1028 Acked Write 10 Done Done Sent false
1029 Acked Read 6 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 2 3
Done 10 9 8 27
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1019 1019 1019
Downstairs last five completed: 1019 1018 1017 1016 1015
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:48.657895486Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1246}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 10 10 10 16 16 16 F F F
1 020-039 16 16 16 26 26 26 F F F
2 040-059 16 16 16 27 27 27 F F F
3 060-079 16 16 16 26 26 26 T T T
4 080-099 4 4 4 7 7 7 T T T
5 100-119 13 13 13 20 20 20 T T T
6 120-139 9 9 9 14 14 14 F F F
7 140-159 14 14 14 23 23 23 F F F
8 160-179 14 14 14 21 21 21 T T T
9 180-199 14 14 14 21 21 21 T T T
10 200-219 16 16 16 27 27 27 F F F
11 220-239 14 14 14 23 23 23 F F F
12 240-259 16 16 16 25 25 25 F F F
13 260-279 16 16 16 25 25 25 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 14 14 14 21 21 21 T T T
16 320-339 12 12 12 19 19 19 F F F
17 340-359 15 15 15 24 24 24 F F F
18 360-379 14 14 14 23 23 23 F F F
19 380-399 13 13 13 20 20 20 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 16 16 16 25 25 25 F F F
22 440-459 14 14 14 21 21 21 F F F
23 460-479 10 10 10 16 16 16 F F F
24 480-499 16 16 16 27 27 27 T T T
25 500-519 14 14 14 21 21 21 F F F
26 520-539 14 14 14 21 21 21 T T T
27 540-559 14 14 14 21 21 21 F F F
28 560-579 16 16 16 27 27 27 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 16, Max flush: 27
A Difference in extent metadata is expected here
On loop 8
Verifying data now
No change after verify
Loop: 8 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 10 10 10 16 16 16 F F F
1 020-039 16 16 16 26 26 26 F F F
2 040-059 16 16 16 27 27 27 F F F
3 060-079 17 17 17 28 28 28 F F F
4 080-099 17 17 17 28 28 28 F F F
5 100-119 17 17 17 28 28 28 F F F
6 120-139 9 9 9 14 14 14 F F F
7 140-159 14 14 14 23 23 23 F F F
8 160-179 17 17 17 28 28 28 F F F
9 180-199 17 17 17 28 28 28 F F F
10 200-219 16 16 16 27 27 27 F F F
11 220-239 14 14 14 23 23 23 F F F
12 240-259 16 16 16 25 25 25 F F F
13 260-279 16 16 16 25 25 25 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 17 17 17 28 28 28 F F F
16 320-339 12 12 12 19 19 19 F F F
17 340-359 15 15 15 24 24 24 F F F
18 360-379 14 14 14 23 23 23 F F F
19 380-399 13 13 13 20 20 20 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 16 16 16 25 25 25 F F F
22 440-459 14 14 14 21 21 21 F F F
23 460-479 10 10 10 16 16 16 F F F
24 480-499 17 17 17 28 28 28 F F F
25 500-519 14 14 14 21 21 21 F F F
26 520-539 17 17 17 28 28 28 F F F
27 540-559 14 14 14 21 21 21 F F F
28 560-579 16 16 16 27 27 27 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 17, Max flush: 28
Begin loop 9
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.928404512Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.97409171Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.974976932Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.975019625Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.975038723Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.975053614Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254}
{"msg":"Crucible a34b19f4-e441-458c-9edb-4341a494c74c has session id: 22569567-485e-47b9-8e68-ad4c350cd9b7","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.975675609Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"Upstairs opts: Upstairs UUID: a34b19f4-e441-458c-9edb-4341a494c74c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.975710078Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"Crucible stats registered with UUID: a34b19f4-e441-458c-9edb-4341a494c74c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.975725138Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:52.97573837Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.976234953Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"task":"crutest"}
{"msg":"The guest has requested activation with gen:18","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.976259747Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254}
{"msg":"a34b19f4-e441-458c-9edb-4341a494c74c active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.976328784Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.976345274Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"0","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.976369499Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"0","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.976382751Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"1","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.976399281Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"1","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.976423126Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"2","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.97644863Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"2","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.976470447Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"io task","client":"2","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.976496591Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"io task","client":"0","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.976536876Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"io task","client":"1","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"ds_connection connected from Ok(127.0.0.1:59220)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.976564629Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"io task","client":"2","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49074)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.976588334Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"io task","client":"1","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"ds_connection connected from Ok(127.0.0.1:41651)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.97677131Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"io task","client":"0","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.977308278Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"0","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.977336261Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"2","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.977913563Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"1","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.977940207Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"0","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.977962303Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"2","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.97854916Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.978574494Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"1","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.978591024Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"[0]R flush_numbers[0..12]: [16, 26, 27, 28, 28, 28, 14, 23, 28, 28, 27, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.97914541Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"downstairs","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"[0]R generation[0..12]: [10, 16, 16, 17, 17, 17, 9, 14, 17, 17, 16, 14]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979172883Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"downstairs","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979188614Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"downstairs","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"[1]R flush_numbers[0..12]: [16, 26, 27, 28, 28, 28, 14, 23, 28, 28, 27, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979202335Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"downstairs","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"[1]R generation[0..12]: [10, 16, 16, 17, 17, 17, 9, 14, 17, 17, 16, 14]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979215747Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"downstairs","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979229328Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"downstairs","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"[2]R flush_numbers[0..12]: [16, 26, 27, 28, 28, 28, 14, 23, 28, 28, 27, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.97924245Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"downstairs","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"[2]R generation[0..12]: [10, 16, 16, 17, 17, 17, 9, 14, 17, 17, 16, 14]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979256092Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"downstairs","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979270083Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"downstairs","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"Wait for a query_work_queue command to finish before sending IO
msg":"Max found gen is 18","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979290421Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"downstairs","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"Generation requested: 18 >= found:18","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979303702Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"downstairs","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"Next flush: 29","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979316254Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"downstairs","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979328877Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"":"downstairs","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979341769Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.97935641Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"0","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979369521Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"1","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979382264Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"client":"2","":"downstairs","session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979395026Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979406968Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"a34b19f4-e441-458c-9edb-4341a494c74c is now active with session: 22569567-485e-47b9-8e68-ad4c350cd9b7","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979419501Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979431134Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254,"session_id":"22569567-485e-47b9-8e68-ad4c350cd9b7"}
{"msg":"The guest has finished waiting for activation with:18","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:52.979442836Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 164 len 20480 data: 2 2 2 2 2
02/30 Write block 191 len 36864 data: 2 2 2 3 5 5 5 5 4
03/30 Write block 52 len 16384 data: 3 3 3 3
04/30 Read block 196 len 16384
05/30 Write block 279 len 4096 data: 4
06/30 Read block 358 len 20480
07/30 Read block 16 len 36864
08/30 Read block 546 len 12288
09/30 Read block 369 len 28672
10/30 Write block 90 len 24576 data: 4 3 3 3 3 3
11/30 Read block 418 len 40960
12/30 Read block 538 len 36864
13/30 Read block 103 len 16384
14/30 Write block 346 len 4096 data: 4
15/30 Read block 353 len 40960
16/30 Write block 197 len 28672 data: 6 6 5 5 4 4 4
17/30 Read block 117 len 40960
18/30 Read block 495 len 24576
19/30 Write block 116 len 28672 data: 4 4 4 4 3 3 2
20/30 Write block 202 len 16384 data: 5 5 4 5
21/30 Read block 291 len 12288
22/30 Flush
23/30 Write block 442 len 24576 data: 2 2 2 2 2 2
24/30 Read block 153 len 40960
25/30 Read block 82 len 20480
26/30 Read block 514 len 40960
27/30 Write block 105 len 32768 data: 2 2 2 2 3 3 3 3
28/30 Write block 23 len 36864 data: 2 2 2 3 4 5 5 6 5
29/30 Read block 185 len 4096
30/30 Write block 555 len 24576 data: 3 3 3 3 5 4
----------------------------------------------------------------
Crucible gen:18 GIO:true work queues: Upstairs:0 downstairs:8
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1022 Acked Write 6 Done Done Done false
1023 Acked Read 10 Done Done Done false
1024 Acked Read 5 Done Done Done false
1025 Acked Read 10 Done Done Done false
1026 Acked Write 8 Done Done Done false
1027 Acked Write 9 Done Done Done false
1028 Acked Read 1 Done Done Done false
1029 Acked Write 6 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 7 7 7 21
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1021 1021 1021
Downstairs last five completed: 1021 1020 1019 1018 1017
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:53.000232522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1254}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 10 10 10 16 16 16 F F F
1 020-039 16 16 16 26 26 26 T T T
2 040-059 18 18 18 29 29 29 F F F
3 060-079 17 17 17 28 28 28 F F F
4 080-099 18 18 18 29 29 29 F F F
5 100-119 18 18 18 29 29 29 T T T
6 120-139 18 18 18 29 29 29 F F F
7 140-159 14 14 14 23 23 23 F F F
8 160-179 18 18 18 29 29 29 F F F
9 180-199 18 18 18 29 29 29 F F F
10 200-219 18 18 18 29 29 29 F F F
11 220-239 14 14 14 23 23 23 F F F
12 240-259 16 16 16 25 25 25 F F F
13 260-279 18 18 18 29 29 29 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 17 17 17 28 28 28 F F F
16 320-339 12 12 12 19 19 19 F F F
17 340-359 18 18 18 29 29 29 F F F
18 360-379 14 14 14 23 23 23 F F F
19 380-399 13 13 13 20 20 20 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 16 16 16 25 25 25 F F F
22 440-459 14 14 14 21 21 21 T T T
23 460-479 10 10 10 16 16 16 F F F
24 480-499 17 17 17 28 28 28 F F F
25 500-519 14 14 14 21 21 21 F F F
26 520-539 17 17 17 28 28 28 F F F
27 540-559 14 14 14 21 21 21 T T T
28 560-579 16 16 16 27 27 27 T T T
29 580-599 11 11 11 17 17 17 F F F
Max gen: 18, Max flush: 29
A Difference in extent metadata is expected here
On loop 9
Verifying data now
No change after verify
Loop: 9 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 10 10 10 16 16 16 F F F
1 020-039 19 19 19 30 30 30 F F F
2 040-059 18 18 18 29 29 29 F F F
3 060-079 17 17 17 28 28 28 F F F
4 080-099 18 18 18 29 29 29 F F F
5 100-119 19 19 19 30 30 30 F F F
6 120-139 18 18 18 29 29 29 F F F
7 140-159 14 14 14 23 23 23 F F F
8 160-179 18 18 18 29 29 29 F F F
9 180-199 18 18 18 29 29 29 F F F
10 200-219 18 18 18 29 29 29 F F F
11 220-239 14 14 14 23 23 23 F F F
12 240-259 16 16 16 25 25 25 F F F
13 260-279 18 18 18 29 29 29 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 17 17 17 28 28 28 F F F
16 320-339 12 12 12 19 19 19 F F F
17 340-359 18 18 18 29 29 29 F F F
18 360-379 14 14 14 23 23 23 F F F
19 380-399 13 13 13 20 20 20 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 16 16 16 25 25 25 F F F
22 440-459 19 19 19 30 30 30 F F F
23 460-479 10 10 10 16 16 16 F F F
24 480-499 17 17 17 28 28 28 F F F
25 500-519 14 14 14 21 21 21 F F F
26 520-539 17 17 17 28 28 28 F F F
27 540-559 19 19 19 30 30 30 F F F
28 560-579 19 19 19 30 30 30 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 19, Max flush: 30
Begin loop 10
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.251536851Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.297234952Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.29845066Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.298497331Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.298512161Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.298527942Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262}
{"msg":"Crucible 624ee2cc-2b46-48b9-ab22-5f299cde23cf has session id: 709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.299267354Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"Upstairs opts: Upstairs UUID: 624ee2cc-2b46-48b9-ab22-5f299cde23cf, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.29945064Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"Crucible stats registered with UUID: 624ee2cc-2b46-48b9-ab22-5f299cde23cf","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.299496632Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:57.299520687Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.299940547Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"task":"crutest"}
{"msg":"The guest has requested activation with gen:20","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.299965781Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262}
{"msg":"624ee2cc-2b46-48b9-ab22-5f299cde23cf active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.300037527Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.300065789Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"0","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.300100028Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"0","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.300122804Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"1","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.300151316Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"1","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.300175192Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"2","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.300201885Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"2","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.300228449Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"io task","client":"2","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.300251085Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"io task","client":"0","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.300273491Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"io task","client":"1","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50271)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.300297516Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"io task","client":"0","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53639)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.30032246Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"io task","client":"1","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:55785)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.300348185Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"io task","client":"2","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.301069738Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"1","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.301109653Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"0","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.30117924Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"2","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.301785125Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"1","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.301830567Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"2","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.301862237Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"0","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302415085Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302533052Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"[0]R flush_numbers[0..12]: [16, 30, 29, 28, 29, 30, 29, 23, 29, 29, 29, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302593754Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"downstairs","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"[0]R generation[0..12]: [10, 19, 18, 17, 18, 19, 18, 14, 18, 18, 18, 14]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302621317Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"downstairs","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302636907Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"downstairs","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"[1]R flush_numbers[0..12]: [16, 30, 29, 28, 29, 30, 29, 23, 29, 29, 29, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302650899Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"downstairs","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"[1]R generation[0..12]: [10, 19, 18, 17, 18, 19, 18, 14, 18, 18, 18, 14]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.30266427Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"downstairs","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302679631Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"downstairs","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"[2]R flush_numbers[0..12]: [16, 30, 29, 28, 29, 30, 29, 23, 29, 29, 29, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302693552Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"downstairs","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"[2]R generation[0..12]: [10, 19, 18, 17, 18, 19, 18, 14, 18, 18, 18, 14]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302707284Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"downstairs","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302720965Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"downstairs","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"Max found gen is 20","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302738374Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"downstairs","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"Generation requested: 20 >= found:20","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302751386Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"downstairs","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"Next flush: 31","v":0,"name":"crucible","level":Wait for a query_work_queue command to finish before sending IO
30,"time":"2025-01-27T19:36:57.302782767Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"downstairs","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.30280921Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"":"downstairs","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302833535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302846657Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"0","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302861188Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"1","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302883754Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"client":"2","":"downstairs","session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302908339Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302931865Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"624ee2cc-2b46-48b9-ab22-5f299cde23cf is now active with session: 709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302945686Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302957509Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262,"session_id":"709494e7-6a5d-49a3-8cc6-f4f2d99e2d0c"}
{"msg":"The guest has finished waiting for activation with:20","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:36:57.302976957Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Flush
02/30 Write block 100 len 28672 data: 2 2 3 2 2 3 3
03/30 Read block 415 len 32768
04/30 Read block 481 len 16384
05/30 Read block 553 len 8192
06/30 Read block 566 len 40960
07/30 Write block 79 len 24576 data: 3 3 3 3 3 3
08/30 Read block 597 len 12288
09/30 Flush
10/30 Write block 349 len 20480 data: 5 4 3 4 4
11/30 Read block 456 len 16384
12/30 Read block 340 len 24576
13/30 Read block 548 len 28672
14/30 Flush
15/30 Write block 474 len 32768 data: 3 3 3 3 3 2 3 3
16/30 Read block 509 len 24576
17/30 Read block 465 len 36864
18/30 Read block 353 len 20480
19/30 Flush
20/30 Write block 260 len 16384 data: 2 2 2 2
21/30 Read block 442 len 40960
22/30 Write block 134 len 36864 data: 3 3 3 3 3 3 3 2 2
23/30 Write block 74 len 4096 data: 3
24/30 Write block 370 len 40960 data: 3 3 3 3 3 3 3 4 4 4
25/30 Flush
26/30 Write block 335 len 36864 data: 2 2 2 2 3 3 3 3 3
27/30 Read block 408 len 20480
28/30 Write block 243 len 28672 data: 2 2 2 2 2 2 2
29/30 Read block 282 len 28672
30/30 Read block 351 len 36864
----------------------------------------------------------------
Crucible gen:20 GIO:true work queues: Upstairs:0 downstairs:5
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1025 Acked Write 9 Done Done Done false
1026 Acked Read 5 Done Done Done false
1027 Acked Write 7 Done Done Done false
1028 Acked Read 7 Done Done Done false
1029 Acked Read 9 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 1 2
Done 5 4 4 13
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1024 1024 1024
Downstairs last five completed: 1024 1023 1022 1021 1020
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:36:57.325110371Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1262}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 10 10 10 16 16 16 F F F
1 020-039 19 19 19 30 30 30 F F F
2 040-059 18 18 18 29 29 29 F F F
3 060-079 20 20 20 35 35 35 F F F
4 080-099 20 20 20 32 32 32 F F F
5 100-119 20 20 20 32 32 32 F F F
6 120-139 20 20 20 35 35 35 F F F
7 140-159 20 20 20 35 35 35 F F F
8 160-179 18 18 18 29 29 29 F F F
9 180-199 18 18 18 29 29 29 F F F
10 200-219 18 18 18 29 29 29 F F F
11 220-239 14 14 14 23 23 23 F F F
12 240-259 16 16 16 25 25 25 T T T
13 260-279 20 20 20 35 35 35 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 17 17 17 28 28 28 F F F
16 320-339 12 12 12 19 19 19 T T T
17 340-359 20 20 20 33 33 33 T T T
18 360-379 20 20 20 35 35 35 F F F
19 380-399 13 13 13 20 20 20 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 16 16 16 25 25 25 F F F
22 440-459 19 19 19 30 30 30 F F F
23 460-479 20 20 20 34 34 34 F F F
24 480-499 20 20 20 34 34 34 F F F
25 500-519 14 14 14 21 21 21 F F F
26 520-539 17 17 17 28 28 28 F F F
27 540-559 19 19 19 30 30 30 F F F
28 560-579 19 19 19 30 30 30 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 20, Max flush: 35
A Difference in extent metadata is expected here
On loop 10
Verifying data now
No change after verify
Loop: 10 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 10 10 10 16 16 16 F F F
1 020-039 19 19 19 30 30 30 F F F
2 040-059 18 18 18 29 29 29 F F F
3 060-079 20 20 20 35 35 35 F F F
4 080-099 20 20 20 32 32 32 F F F
5 100-119 20 20 20 32 32 32 F F F
6 120-139 20 20 20 35 35 35 F F F
7 140-159 20 20 20 35 35 35 F F F
8 160-179 18 18 18 29 29 29 F F F
9 180-199 18 18 18 29 29 29 F F F
10 200-219 18 18 18 29 29 29 F F F
11 220-239 14 14 14 23 23 23 F F F
12 240-259 21 21 21 36 36 36 F F F
13 260-279 20 20 20 35 35 35 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 17 17 17 28 28 28 F F F
16 320-339 21 21 21 36 36 36 F F F
17 340-359 21 21 21 36 36 36 F F F
18 360-379 20 20 20 35 35 35 F F F
19 380-399 13 13 13 20 20 20 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 16 16 16 25 25 25 F F F
22 440-459 19 19 19 30 30 30 F F F
23 460-479 20 20 20 34 34 34 F F F
24 480-499 20 20 20 34 34 34 F F F
25 500-519 14 14 14 21 21 21 F F F
26 520-539 17 17 17 28 28 28 F F F
27 540-559 19 19 19 30 30 30 F F F
28 560-579 19 19 19 30 30 30 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 21, Max flush: 36
Begin loop 11
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.570201646Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:01.613742484Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.614417466Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.6464662Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.647570956Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.647609302Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.647623743Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.647638374Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270}
{"msg":"Crucible 21dbd4f1-e47d-4225-8f03-3428b9b16eb9 has session id: 77f51c67-acda-41ef-9b53-8513e83d14f1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.648362745Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"Upstairs opts: Upstairs UUID: 21dbd4f1-e47d-4225-8f03-3428b9b16eb9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.648398393Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"Crucible stats registered with UUID: 21dbd4f1-e47d-4225-8f03-3428b9b16eb9","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.648418511Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:01.648431493Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.648899643Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"task":"crutest"}
{"msg":"The guest has requested activation with gen:22","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.648929385Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270}
{"msg":"21dbd4f1-e47d-4225-8f03-3428b9b16eb9 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.649002849Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.649020368Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"0","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.649046302Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"0","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.649061763Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"1","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.649076614Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"1","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.649097321Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"2","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.649126962Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"2","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.649154465Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"io task","client":"2","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.649182168Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"io task","client":"0","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.649207712Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"io task","client":"1","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"ds_connection connected from Ok(127.0.0.1:54426)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.649232697Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"io task","client":"2","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"ds_connection connected from Ok(127.0.0.1:60361)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.649256702Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"io task","client":"1","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50775)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.649286663Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"io task","client":"0","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.650101629Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"2","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.650137686Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"1","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.650393248Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"0","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.650897206Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"1","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651026196Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"2","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651067521Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"0","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651518811Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651543366Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"[0]R flush_numbers[0..12]: [16, 30, 29, 35, 32, 32, 35, 35, 29, 29, 29, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651583841Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"downstairs","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"[0]R generation[0..12]: [10, 19, 18, 20, 20, 20, 20, 20, 18, 18, 18, 14]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.65160173Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"downstairs","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651621478Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"downstairs","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"[1]R flush_numbers[0..12]: [16, 30, 29, 35, 32, 32, 35, 35, 29, 29, 29, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651636398Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"downstairs","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"[1]R generation[0..12]: [10, 19, 18, 20, 20, 20, 20, 20, 18, 18, 18, 14]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.65165034Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"downstairs","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651667569Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"downstairs","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"[2]R flush_numbers[0..12]: [16, 30, 29, 35, 32, 32, 35, 35, 29, 29, 29, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.65168143Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"downstairs","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"[2]R generation[0..12]: [10, 19, 18, 20, 20, 20, 20, 20, 18, 18, 18, 14]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651694692Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"downstairs","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651708184Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"downstairs","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"Max found gen is 22","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651720726Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"downstairs","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"Generation requested: 22 >= found:22","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:37:01.651733708Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"downstairs","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"Next flush: 37","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651748519Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"downstairs","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651761301Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"":"downstairs","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651773903Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651785586Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"0","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651798598Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"1","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.65181118Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"client":"2","":"downstairs","session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651824022Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651835965Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"21dbd4f1-e47d-4225-8f03-3428b9b16eb9 is now active with session: 77f51c67-acda-41ef-9b53-8513e83d14f1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651848437Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651861489Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270,"session_id":"77f51c67-acda-41ef-9b53-8513e83d14f1"}
{"msg":"The guest has finished waiting for activation with:22","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:01.651873571Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 530 len 28672 data: 7 7 6 6 6 6 4
02/30 Read block 461 len 20480
03/30 Write block 429 len 32768 data: 2 2 2 2 2 3 3 3
04/30 Write block 336 len 36864 data: 3 3 3 4 4 4 4 4 4
05/30 Read block 233 len 20480
06/30 Read block 356 len 12288
07/30 Read block 443 len 28672
08/30 Write block 136 len 32768 data: 4 4 4 4 4 3 3 3
09/30 Write block 22 len 12288 data: 3 3 3
10/30 Flush
11/30 Write block 478 len 4096 data: 4
12/30 Read block 173 len 20480
13/30 Write block 153 len 16384 data: 5 4 2 2
14/30 Read block 87 len 16384
15/30 Write block 314 len 24576 data: 3 2 2 2 2 3
16/30 Flush
17/30 Write block 239 len 28672 data: 2 2 2 2 3 3 3
18/30 Write block 170 len 12288 data: 2 3 3
19/30 Write block 150 len 20480 data: 3 3 3 6 5
20/30 Write block 391 len 16384 data: 3 4 3 3
21/30 Read block 6 len 40960
22/30 Write block 92 len 4096 data: 4
23/30 Write block 448 len 24576 data: 2 2 2 2 2 2
24/30 Read block 574 len 24576
25/30 Write block 5 len 4096 data: 2
26/30 Read block 383 len 20480
27/30 Read block 41 len 36864
28/30 Read block 18 len 24576
29/30 Write block 526 len 12288 data: 4 7 8
30/30 Read block 361 len 24576
----------------------------------------------------------------
Crucible gen:22 GIO:true work queues: Upstairs:0 downstairs:14
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1016 Acked Write 7 Done Done Done false
1017 Acked Write 3 Done Done Done false
1018 Acked Write 5 Done Done Done false
1019 Acked Write 4 Done Done Done false
1020 Acked Read 10 Done Done Done false
1021 Acked Write 1 Done Done Done false
1022 Acked Write 6 Done Done Done false
1023 Acked Read 6 Done Done Done false
1024 Acked Write 1 Done Done Done false
1025 Acked Read 5 Done Done Done false
1026 Acked Read 9 Done Done Done false
1027 Acked Read 6 Done Done Done false
1028 Acked Write 3 Done Done Done false
1029 Acked Read 6 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 14 14 14 42
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1015 1015 1015
Downstairs last five completed: 1015 1014 1013 1012 1011
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:01.672270641Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1270}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 10 10 10 16 16 16 T T T
1 020-039 22 22 22 37 37 37 F F F
2 040-059 18 18 18 29 29 29 F F F
3 060-079 20 20 20 35 35 35 F F F
4 080-099 20 20 20 32 32 32 T T T
5 100-119 20 20 20 32 32 32 F F F
6 120-139 22 22 22 37 37 37 F F F
7 140-159 22 22 22 38 38 38 T T T
8 160-179 18 18 18 29 29 29 T T T
9 180-199 18 18 18 29 29 29 F F F
10 200-219 18 18 18 29 29 29 F F F
11 220-239 14 14 14 23 23 23 T T T
12 240-259 21 21 21 36 36 36 T T T
13 260-279 20 20 20 35 35 35 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 22 22 22 38 38 38 F F F
16 320-339 22 22 22 37 37 37 F F F
17 340-359 22 22 22 37 37 37 F F F
18 360-379 20 20 20 35 35 35 F F F
19 380-399 13 13 13 20 20 20 T T T
20 400-419 15 15 15 24 24 24 F F F
21 420-439 22 22 22 37 37 37 F F F
22 440-459 19 19 19 30 30 30 T T T
23 460-479 22 22 22 38 38 38 F F F
24 480-499 20 20 20 34 34 34 F F F
25 500-519 14 14 14 21 21 21 F F F
26 520-539 22 22 22 37 37 37 T T T
27 540-559 19 19 19 30 30 30 F F F
28 560-579 19 19 19 30 30 30 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 22, Max flush: 38
A Difference in extent metadata is expected here
On loop 11
Verifying data now
No change after verify
Loop: 11 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 23 23 23 39 39 39 F F F
1 020-039 22 22 22 37 37 37 F F F
2 040-059 18 18 18 29 29 29 F F F
3 060-079 20 20 20 35 35 35 F F F
4 080-099 23 23 23 39 39 39 F F F
5 100-119 20 20 20 32 32 32 F F F
6 120-139 22 22 22 37 37 37 F F F
7 140-159 23 23 23 39 39 39 F F F
8 160-179 23 23 23 39 39 39 F F F
9 180-199 18 18 18 29 29 29 F F F
10 200-219 18 18 18 29 29 29 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 20 20 20 35 35 35 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 22 22 22 38 38 38 F F F
16 320-339 22 22 22 37 37 37 F F F
17 340-359 22 22 22 37 37 37 F F F
18 360-379 20 20 20 35 35 35 F F F
19 380-399 23 23 23 39 39 39 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 22 22 22 37 37 37 F F F
22 440-459 23 23 23 39 39 39 F F F
23 460-479 22 22 22 38 38 38 F F F
24 480-499 20 20 20 34 34 34 F F F
25 500-519 14 14 14 21 21 21 F F F
26 520-539 23 23 23 39 39 39 F F F
27 540-559 19 19 19 30 30 30 F F F
28 560-579 19 19 19 30 30 30 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 23, Max flush: 39
Begin loop 12
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:05.985300625Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:06.028802497Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.029448547Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.061636415Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.062738842Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.062782715Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.062799385Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.062816964Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278}
{"msg":"Crucible ea376451-fc83-4903-93e7-3f1597815919 has session id: 820e2a1d-0e3f-4e23-a67d-a099148293f3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.063482062Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"Upstairs opts: Upstairs UUID: ea376451-fc83-4903-93e7-3f1597815919, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.063528963Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"Crucible stats registered with UUID: ea376451-fc83-4903-93e7-3f1597815919","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.06354994Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:06.06356664Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.063941098Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"task":"crutest"}
{"msg":"The guest has requested activation with gen:24","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.063976386Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278}
{"msg":"ea376451-fc83-4903-93e7-3f1597815919 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.064047842Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.064074605Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"0","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.064102378Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"0","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.064118358Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"1","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.064133449Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"1","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.064168497Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"2","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.064184887Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"2","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.064198498Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"io task","client":"2","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.064304752Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"io task","client":"0","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.064335264Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"io task","client":"1","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52453)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.064361008Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"io task","client":"0","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39120)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.064385282Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"io task","client":"1","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"ds_connection connected from Ok(127.0.0.1:46017)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.064408748Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"io task","client":"2","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.065081062Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"1","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.065118339Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"2","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.065264308Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"0","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.065833106Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"1","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.065870663Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"2","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.065897856Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"0","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066446416Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066558737Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"[0]R flush_numbers[0..12]: [39, 37, 29, 35, 39, 32, 37, 39, 39, 29, 29, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066592406Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"downstairs","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"[0]R generation[0..12]: [23, 22, 18, 20, 23, 20, 22, 23, 23, 18, 18, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066607686Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"downstairs","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066621937Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"downstairs","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"[1]R flush_numbers[0..12]: [39, 37, 29, 35, 39, 32, 37, 39, 39, 29, 29, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066635699Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"downstairs","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"[1]R generation[0..12]: [23, 22, 18, 20, 23, 20, 22, 23, 23, 18, 18, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066649031Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"downstairs","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066662432Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"downstairs","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"[2]R flush_numbers[0..12]: [39, 37, 29, 35, 39, 32, 37, 39, 39, 29, 29, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066675724Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"downstairs","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"[2]R generation[0..12]: [23, 22, 18, 20, 23, 20, 22, 23, 23, 18, 18, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066689246Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"downstairs","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066702967Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"downstairs","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"Max found gen is 24","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.06671555Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"downstairs","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"Generation requested: 24 >= found:24","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066733568Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"downstairs","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"Next flush: 40","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066756534Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"downstairs","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"All extents match","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:37:06.066781239Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"":"downstairs","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066801337Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066813149Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"0","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066826231Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"1","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066839313Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"client":"2","":"downstairs","session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066852405Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066864417Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"ea376451-fc83-4903-93e7-3f1597815919 is now active with session: 820e2a1d-0e3f-4e23-a67d-a099148293f3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.06687689Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066888732Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278,"session_id":"820e2a1d-0e3f-4e23-a67d-a099148293f3"}
{"msg":"The guest has finished waiting for activation with:24","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:06.066900655Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 324 len 24576 data: 5 5 4 4 3 2
02/30 Read block 450 len 16384
03/30 Write block 557 len 28672 data: 4 4 6 5 3 3 3
04/30 Read block 501 len 4096
05/30 Read block 61 len 20480
06/30 Flush
07/30 Write block 369 len 4096 data: 3
08/30 Write block 503 len 40960 data: 3 3 2 3 3 3 3 3 3 3
09/30 Write block 469 len 8192 data: 2 2
10/30 Flush
11/30 Write block 515 len 36864 data: 2 2 2 2 2 3 4 5 5
12/30 Write block 563 len 12288 data: 4 3 4
13/30 Read block 168 len 4096
14/30 Write block 323 len 32768 data: 5 6 6 5 5 4 3 2
15/30 Write block 355 len 24576 data: 4 5 4 4 4 4
16/30 Read block 70 len 36864
17/30 Flush
18/30 Write block 181 len 28672 data: 6 5 5 4 3 3 3
19/30 Read block 232 len 20480
20/30 Read block 512 len 24576
21/30 Read block 46 len 40960
22/30 Read block 144 len 24576
23/30 Write block 195 len 16384 data: 6 6 7 7
24/30 Read block 161 len 28672
25/30 Flush
26/30 Write block 460 len 20480 data: 3 3 2 3 2
27/30 Read block 68 len 40960
28/30 Write block 443 len 40960 data: 3 3 3 3 3 3 3 3 3 3
29/30 Write block 344 len 8192 data: 5 4
30/30 Read block 103 len 24576
----------------------------------------------------------------
Crucible gen:24 GIO:true work queues: Upstairs:0 downstairs:5
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1025 Acked Write 5 Done Done Done false
1026 Acked Read 10 Done Done Done false
1027 Acked Write 10 Done Done Done false
1028 Acked Write 2 Done Done Done false
1029 Acked Read 6 Sent Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 0 1 2
Done 4 5 4 13
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1024 1024 1024
Downstairs last five completed: 1024 1023 1022 1021 1020
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:06.087883521Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1278}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 23 23 23 39 39 39 F F F
1 020-039 22 22 22 37 37 37 F F F
2 040-059 18 18 18 29 29 29 F F F
3 060-079 20 20 20 35 35 35 F F F
4 080-099 23 23 23 39 39 39 F F F
5 100-119 20 20 20 32 32 32 F F F
6 120-139 22 22 22 37 37 37 F F F
7 140-159 23 23 23 39 39 39 F F F
8 160-179 23 23 23 39 39 39 F F F
9 180-199 24 24 24 43 43 43 F F F
10 200-219 18 18 18 29 29 29 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 20 20 20 35 35 35 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 22 22 22 38 38 38 F F F
16 320-339 24 24 24 42 42 42 F F F
17 340-359 24 24 24 42 42 42 T T T
18 360-379 24 24 24 42 42 42 F F F
19 380-399 23 23 23 39 39 39 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 22 22 22 37 37 37 F F F
22 440-459 23 23 23 39 39 39 T T T
23 460-479 24 24 24 41 41 41 T T T
24 480-499 20 20 20 34 34 34 F F F
25 500-519 24 24 24 42 42 42 F F F
26 520-539 24 24 24 42 42 42 F F F
27 540-559 24 24 24 40 40 40 F F F
28 560-579 24 24 24 42 42 42 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 24, Max flush: 43
A Difference in extent metadata is expected here
On loop 12
Verifying data now
No change after verify
Loop: 12 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 23 23 23 39 39 39 F F F
1 020-039 22 22 22 37 37 37 F F F
2 040-059 18 18 18 29 29 29 F F F
3 060-079 20 20 20 35 35 35 F F F
4 080-099 23 23 23 39 39 39 F F F
5 100-119 20 20 20 32 32 32 F F F
6 120-139 22 22 22 37 37 37 F F F
7 140-159 23 23 23 39 39 39 F F F
8 160-179 23 23 23 39 39 39 F F F
9 180-199 24 24 24 43 43 43 F F F
10 200-219 18 18 18 29 29 29 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 20 20 20 35 35 35 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 22 22 22 38 38 38 F F F
16 320-339 24 24 24 42 42 42 F F F
17 340-359 25 25 25 44 44 44 F F F
18 360-379 24 24 24 42 42 42 F F F
19 380-399 23 23 23 39 39 39 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 22 22 22 37 37 37 F F F
22 440-459 25 25 25 44 44 44 F F F
23 460-479 25 25 25 44 44 44 F F F
24 480-499 20 20 20 34 34 34 F F F
25 500-519 24 24 24 42 42 42 F F F
26 520-539 24 24 24 42 42 42 F F F
27 540-559 24 24 24 40 40 40 F F F
28 560-579 24 24 24 42 42 42 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 25, Max flush: 44
Begin loop 13
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.369365171Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.415283566Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.416361179Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.41640769Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.416425159Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.41643979Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286}
{"msg":"Crucible 2ac43259-f038-41ff-b9d9-f4d4c3b8c3c4 has session id: f15acd78-ec82-483b-8a39-1e8e99e52454","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.41711938Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"Upstairs opts: Upstairs UUID: 2ac43259-f038-41ff-b9d9-f4d4c3b8c3c4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417165611Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"Crucible stats registered with UUID: 2ac43259-f038-41ff-b9d9-f4d4c3b8c3c4","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417184919Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:10.417197651Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417661884Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"task":"crutest"}
{"msg":"The guest has requested activation with gen:26","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417697102Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286}
{"msg":"2ac43259-f038-41ff-b9d9-f4d4c3b8c3c4 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417724145Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417752208Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"0","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417788495Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"0","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417803546Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"1","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417817397Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"1","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417830519Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"2","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417853555Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"2","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417880398Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"io task","client":"2","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417907732Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"io task","client":"0","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417935724Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"io task","client":"1","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53752)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417966885Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"io task","client":"2","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47136)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.417994108Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"io task","client":"0","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"ds_connection connected from Ok(127.0.0.1:54910)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.418021931Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"io task","client":"1","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.41866827Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"2","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.418697312Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"0","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.419397739Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"0","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.419435785Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"2","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.420070512Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.420099264Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.420508811Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"1","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421081566Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"1","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"[0]R flush_numbers[0..12]: [39, 37, 29, 35, 39, 32, 37, 39, 39, 43, 29, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421642179Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"downstairs","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"[0]R generation[0..12]: [23, 22, 18, 20, 23, 20, 22, 23, 23, 24, 18, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421662606Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"downstairs","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421676917Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"downstairs","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"[1]R flush_numbers[0..12]: [39, 37, 29, 35, 39, 32, 37, 39, 39, 43, 29, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421690939Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"downstairs","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"[1]R generation[0..12]: [23, 22, 18, 20, 23, 20, 22, 23, 23, 24, 18, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.42170427Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"downstairs","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421718282Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"downstairs","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"[2]R flush_numbers[0..12]: [39, 37, 29, 35, 39, 32, 37, 39, 39, 43, 29, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421731214Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"downstairs","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"[2]R generation[0..12]: [23, 22, 18, 20, 23, 20, 22, 23, 23, 24, 18, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421744116Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"downstairs","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421760875Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"downstairs","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"Max found gen is 26","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421773488Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"downstairs","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"Generation requested: 26 >= found:26","v":0,"name":"crucible","level":30,"time"Wait for a query_work_queue command to finish before sending IO
:"2025-01-27T19:37:10.42178617Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"downstairs","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"Next flush: 45","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421806867Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"downstairs","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421819379Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"":"downstairs","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421831831Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421843454Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"0","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421856396Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"1","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421868939Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"client":"2","":"downstairs","session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.42188253Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421894433Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"2ac43259-f038-41ff-b9d9-f4d4c3b8c3c4 is now active with session: f15acd78-ec82-483b-8a39-1e8e99e52454","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421906795Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421918278Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286,"session_id":"f15acd78-ec82-483b-8a39-1e8e99e52454"}
{"msg":"The guest has finished waiting for activation with:26","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:10.421930121Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 449 len 20480 data: 4 4 4 4 3
02/30 Write block 495 len 8192 data: 3 4
03/30 Read block 411 len 28672
04/30 Read block 117 len 28672
05/30 Write block 353 len 8192 data: 5 4
06/30 Read block 93 len 8192
07/30 Write block 511 len 32768 data: 4 4 2 2 3 3 3 3
08/30 Write block 423 len 28672 data: 4 3 3 2 2 2 3
09/30 Write block 484 len 36864 data: 3 3 3 3 3 2 2 2 4
10/30 Read block 277 len 8192
11/30 Read block 417 len 24576
12/30 Read block 111 len 12288
13/30 Write block 305 len 20480 data: 4 4 4 4 4
14/30 Read block 363 len 28672
15/30 Read block 249 len 4096
16/30 Read block 580 len 32768
17/30 Read block 62 len 28672
18/30 Read block 479 len 32768
19/30 Read block 526 len 20480
20/30 Write block 2 len 36864 data: 2 2 2 3 2 3 3 3 3
21/30 Write block 570 len 20480 data: 3 3 4 4 4
22/30 Write block 34 len 36864 data: 4 4 3 3 3 2 3 3 3
23/30 Read block 443 len 28672
24/30 Read block 63 len 24576
25/30 Read block 376 len 32768
26/30 Read block 160 len 24576
27/30 Write block 208 len 32768 data: 5 4 4 4 4 3 3 3
28/30 Write block 164 len 36864 data: 3 3 3 3 3 2 3 4 4
29/30 Read block 9 len 36864
30/30 Write block 332 len 8192 data: 2 2
----------------------------------------------------------------
Crucible gen:26 GIO:true work queues: Upstairs:0 downstairs:30
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Write 5 Done Done Done false
1001 Acked Write 2 Done Done Done false
1002 Acked Read 7 Done Done Done false
1003 Acked Read 7 Done Done Done false
1004 Acked Write 2 Done Done Done false
1005 Acked Read 2 Done Done Done false
1006 Acked Write 8 Done Done Done false
1007 Acked Write 7 Done Done Done false
1008 Acked Write 9 Done Done Done false
1009 Acked Read 2 Done Done Done false
1010 Acked Read 6 Done Done Done false
1011 Acked Read 3 Done Done Done false
1012 Acked Write 5 Done Done Done false
1013 Acked Read 7 Done Done Done false
1014 Acked Read 1 Done Done Done false
1015 Acked Read 8 Done Done Done false
1016 Acked Read 7 Done Done Done false
1017 Acked Read 8 Done Done Done false
1018 Acked Read 5 Done Done Done false
1019 Acked Write 9 Done Done Done false
1020 Acked Write 5 Done Done Done false
1021 Acked Write 9 Done Done Done false
1022 Acked Read 7 Done Done Done false
1023 Acked Read 6 Done Done Done false
1024 Acked Read 8 Done Done Done false
1025 Acked Read 6 Done Done Done false
1026 Acked Write 8 Done Done Done false
1027 Acked Write 9 Done Done Done false
1028 Acked Read 9 Sent Done Done false
1029 Acked Write 2 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 1 1 4
Done 28 29 29 86
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 0 0 0
Downstairs last five completed:
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:10.442438451Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1286}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 23 23 23 39 39 39 T T T
1 020-039 22 22 22 37 37 37 T T T
2 040-059 18 18 18 29 29 29 T T T
3 060-079 20 20 20 35 35 35 F F F
4 080-099 23 23 23 39 39 39 F F F
5 100-119 20 20 20 32 32 32 F F F
6 120-139 22 22 22 37 37 37 F F F
7 140-159 23 23 23 39 39 39 F F F
8 160-179 23 23 23 39 39 39 T T T
9 180-199 24 24 24 43 43 43 F F F
10 200-219 18 18 18 29 29 29 T T T
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 20 20 20 35 35 35 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 22 22 22 38 38 38 T T T
16 320-339 24 24 24 42 42 42 T T T
17 340-359 25 25 25 44 44 44 T T T
18 360-379 24 24 24 42 42 42 F F F
19 380-399 23 23 23 39 39 39 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 22 22 22 37 37 37 T T T
22 440-459 25 25 25 44 44 44 T T T
23 460-479 25 25 25 44 44 44 F F F
24 480-499 20 20 20 34 34 34 T T T
25 500-519 24 24 24 42 42 42 T T T
26 520-539 24 24 24 42 42 42 F F F
27 540-559 24 24 24 40 40 40 F F F
28 560-579 24 24 24 42 42 42 T T T
29 580-599 11 11 11 17 17 17 F F F
Max gen: 25, Max flush: 44
A Difference in extent metadata is expected here
On loop 13
Verifying data now
No change after verify
Loop: 13 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 26 26 26 45 45 45 F F F
1 020-039 26 26 26 45 45 45 F F F
2 040-059 26 26 26 45 45 45 F F F
3 060-079 20 20 20 35 35 35 F F F
4 080-099 23 23 23 39 39 39 F F F
5 100-119 20 20 20 32 32 32 F F F
6 120-139 22 22 22 37 37 37 F F F
7 140-159 23 23 23 39 39 39 F F F
8 160-179 26 26 26 45 45 45 F F F
9 180-199 24 24 24 43 43 43 F F F
10 200-219 26 26 26 45 45 45 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 20 20 20 35 35 35 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 26 26 26 45 45 45 F F F
16 320-339 26 26 26 45 45 45 F F F
17 340-359 26 26 26 45 45 45 F F F
18 360-379 24 24 24 42 42 42 F F F
19 380-399 23 23 23 39 39 39 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 26 26 26 45 45 45 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 25 25 25 44 44 44 F F F
24 480-499 26 26 26 45 45 45 F F F
25 500-519 26 26 26 45 45 45 F F F
26 520-539 24 24 24 42 42 42 F F F
27 540-559 24 24 24 40 40 40 F F F
28 560-579 26 26 26 45 45 45 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 26, Max flush: 45
Begin loop 14
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.739799937Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.785255569Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.786121363Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.786164516Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.786185014Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.786200804Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294}
{"msg":"Crucible d65fca16-201f-4ceb-bb47-5816a82159da has session id: 4cd8cfbf-61bc-4057-80d5-da2b31475971","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.786914981Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"Upstairs opts: Upstairs UUID: d65fca16-201f-4ceb-bb47-5816a82159da, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.78694946Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"Crucible stats registered with UUID: d65fca16-201f-4ceb-bb47-5816a82159da","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.786968378Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:14.78698206Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787398602Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"task":"crutest"}
{"msg":"The guest has requested activation with gen:28","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787422917Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294}
{"msg":"d65fca16-201f-4ceb-bb47-5816a82159da active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787490405Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787508774Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"0","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787545901Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"0","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787572814Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"1","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787601696Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"1","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787629839Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"2","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787656322Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"2","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787679038Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"io task","client":"2","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787693999Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"io task","client":"0","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787707411Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"io task","client":"1","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53905)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787879274Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"io task","client":"2","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"ds_connection connected from Ok(127.0.0.1:65374)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787922547Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"io task","client":"1","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"ds_connection connected from Ok(127.0.0.1:34508)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.787992084Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"io task","client":"0","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.788623134Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"0","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.788663958Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"1","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.789333364Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"1","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.78937179Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"0","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.789507916Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"2","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.78998576Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790012843Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790254743Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"2","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"[0]R flush_numbers[0..12]: [45, 45, 45, 35, 39, 32, 37, 39, 45, 43, 45, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790782536Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"downstairs","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"[0]R generation[0..12]: [26, 26, 26, 20, 23, 20, 22, 23, 26, 24, 26, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790807831Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"downstairs","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790822922Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"downstairs","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"[1]R flush_numbers[0..12]: [45, 45, 45, 35, 39, 32, 37, 39, 45, 43, 45, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790836443Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"downstairs","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"[1]R generation[0..12]: [26, 26, 26, 20, 23, 20, 22, 23, 26, 24, 26, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790849555Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"downstairs","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790863217Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"downstairs","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"[2]R flush_numbers[0..12]: [45, 45, 45, 35, 39, 32, 37, 39, 45, 43, 45, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790876598Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"downstairs","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"[2]R generation[0..12]: [26, 26, 26, 20, 23, 20, 22, 23, 26, 24, 26, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.79088984Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"downstairs","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790903352Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"downstairs","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"Max found gen is 27","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790915864Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO
4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"Generation requested: 28 >= found:27","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790938919Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"downstairs","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"Next flush: 46","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790962535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"downstairs","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.790985571Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"":"downstairs","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.791007487Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.791033921Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"0","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.791058875Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"1","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
,"time":"2025-01-27T19:37:14.791081701Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"client":"2","":"downstairs","session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.791105826Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.791125354Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"d65fca16-201f-4ceb-bb47-5816a82159da is now active with session: 4cd8cfbf-61bc-4057-80d5-da2b31475971","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.791139825Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.791151668Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294,"session_id":"4cd8cfbf-61bc-4057-80d5-da2b31475971"}
{"msg":"The guest has finished waiting for activation with:28","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:14.79116356Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Flush
02/30 Write block 68 len 28672 data: 5 4 5 5 4 4 4
03/30 Write block 320 len 24576 data: 3 3 4 6 7 7
04/30 Write block 44 len 4096 data: 3
05/30 Flush
06/30 Write block 23 len 16384 data: 4 4 3 4
07/30 Read block 121 len 28672
08/30 Write block 204 len 16384 data: 5 6 4 5
09/30 Flush
10/30 Write block 71 len 12288 data: 6 5 5
11/30 Read block 314 len 16384
12/30 Read block 505 len 24576
13/30 Read block 521 len 24576
14/30 Read block 534 len 24576
15/30 Read block 373 len 28672
16/30 Read block 173 len 8192
17/30 Read block 334 len 36864
18/30 Read block 238 len 16384
19/30 Flush
20/30 Write block 322 len 4096 data: 5
21/30 Flush
22/30 Write block 28 len 24576 data: 6 6 7 6 5 5
23/30 Read block 569 len 36864
24/30 Read block 107 len 12288
25/30 Write block 23 len 16384 data: 5 5 4 5
26/30 Read block 564 len 32768
27/30 Read block 338 len 20480
28/30 Write block 325 len 20480 data: 8 6 6 5 4
29/30 Write block 358 len 4096 data: 5
30/30 Read block 194 len 16384
----------------------------------------------------------------
Crucible gen:28 GIO:true work queues: Upstairs:0 downstairs:9
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1021 Acked Write 6 Done Done Done false
1022 Acked Read 9 Done Done Done false
1023 Acked Read 3 Done Done Done false
1024 Acked Write 4 Done Done Done false
1025 Acked Read 8 Done Done Done false
1026 Acked Read 5 Done Done Done false
1027 Acked Write 5 Done Done Done false
1028 Acked Write 1 Done Done Sent false
1029 Acked Read 4 Sent Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 0 2 3
Done 8 9 7 24
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1020 1020 1020
Downstairs last five completed: 1020 1019 1018 1017 1016
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:14.810407004Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1294}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 26 26 26 45 45 45 F F F
1 020-039 28 28 28 48 48 48 T T T
2 040-059 28 28 28 47 47 47 F F F
3 060-079 28 28 28 49 49 49 F F F
4 080-099 23 23 23 39 39 39 F F F
5 100-119 20 20 20 32 32 32 F F F
6 120-139 22 22 22 37 37 37 F F F
7 140-159 23 23 23 39 39 39 F F F
8 160-179 26 26 26 45 45 45 F F F
9 180-199 24 24 24 43 43 43 F F F
10 200-219 28 28 28 48 48 48 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 20 20 20 35 35 35 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 26 26 26 45 45 45 F F F
16 320-339 28 28 28 50 50 50 T T T
17 340-359 26 26 26 45 45 45 T T T
18 360-379 24 24 24 42 42 42 F F F
19 380-399 23 23 23 39 39 39 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 26 26 26 45 45 45 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 25 25 25 44 44 44 F F F
24 480-499 26 26 26 45 45 45 F F F
25 500-519 26 26 26 45 45 45 F F F
26 520-539 24 24 24 42 42 42 F F F
27 540-559 24 24 24 40 40 40 F F F
28 560-579 26 26 26 45 45 45 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 28, Max flush: 50
A Difference in extent metadata is expected here
On loop 14
Verifying data now
No change after verify
Loop: 14 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 26 26 26 45 45 45 F F F
1 020-039 29 29 29 51 51 51 F F F
2 040-059 28 28 28 47 47 47 F F F
3 060-079 28 28 28 49 49 49 F F F
4 080-099 23 23 23 39 39 39 F F F
5 100-119 20 20 20 32 32 32 F F F
6 120-139 22 22 22 37 37 37 F F F
7 140-159 23 23 23 39 39 39 F F F
8 160-179 26 26 26 45 45 45 F F F
9 180-199 24 24 24 43 43 43 F F F
10 200-219 28 28 28 48 48 48 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 20 20 20 35 35 35 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 26 26 26 45 45 45 F F F
16 320-339 29 29 29 51 51 51 F F F
17 340-359 29 29 29 51 51 51 F F F
18 360-379 24 24 24 42 42 42 F F F
19 380-399 23 23 23 39 39 39 F F F
20 400-419 15 15 15 24 24 24 F F F
21 420-439 26 26 26 45 45 45 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 25 25 25 44 44 44 F F F
24 480-499 26 26 26 45 45 45 F F F
25 500-519 26 26 26 45 45 45 F F F
26 520-539 24 24 24 42 42 42 F F F
27 540-559 24 24 24 40 40 40 F F F
28 560-579 26 26 26 45 45 45 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 29, Max flush: 51
Begin loop 15
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.054618044Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:19.099978435Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.100705764Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.133181914Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.13412417Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.134169482Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.134186352Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.13420434Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302}
{"msg":"Crucible c8e5ffb5-4685-4079-b982-d2ac5a0e4515 has session id: f431f3f6-99da-4751-bf3a-db6f963333d8","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.134819859Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"Upstairs opts: Upstairs UUID: c8e5ffb5-4685-4079-b982-d2ac5a0e4515, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.134861683Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"Crucible stats registered with UUID: c8e5ffb5-4685-4079-b982-d2ac5a0e4515","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.134885938Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:19.134901439Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135309386Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"task":"crutest"}
{"msg":"The guest has requested activation with gen:30","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135332542Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302}
{"msg":"c8e5ffb5-4685-4079-b982-d2ac5a0e4515 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135393144Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135414201Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"0","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135437417Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"0","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135450698Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"1","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135467148Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"1","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135490184Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"2","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135516128Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"2","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135542992Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"io task","client":"2","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135568616Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"io task","client":"0","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135596119Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"io task","client":"1","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"ds_connection connected from Ok(127.0.0.1:35744)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135624721Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"io task","client":"2","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"ds_connection connected from Ok(127.0.0.1:33455)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135655492Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"io task","client":"0","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"ds_connection connected from Ok(127.0.0.1:59114)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.135684264Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"io task","client":"1","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.136397732Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"1","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.13657784Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"2","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.136791748Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"0","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.137018907Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"1","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.137170484Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"2","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.137599408Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.137637384Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"0","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.137678189Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"[0]R flush_numbers[0..12]: [45, 51, 47, 49, 39, 32, 37, 39, 45, 43, 48, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138152596Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"downstairs","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"[0]R generation[0..12]: [26, 29, 28, 28, 23, 20, 22, 23, 26, 24, 28, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138171364Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"downstairs","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138185565Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"downstairs","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"[1]R flush_numbers[0..12]: [45, 51, 47, 49, 39, 32, 37, 39, 45, 43, 48, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138199307Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"downstairs","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"[1]R generation[0..12]: [26, 29, 28, 28, 23, 20, 22, 23, 26, 24, 28, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138212798Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"downstairs","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.13822649Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"downstairs","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"[2]R flush_numbers[0..12]: [45, 51, 47, 49, 39, 32, 37, 39, 45, 43, 48, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138239771Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"downstairs","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"[2]R generation[0..12]: [26, 29, 28, 28, 23, 20, 22, 23, 26, 24, 28, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138253023Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"downstairs","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138266635Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"downstairs","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"Max found gen is 30","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138282335Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"downstairs","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"Generation requested: 30 >= found:30","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138295247Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"downstairs","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"Next flush: 52","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138307629Z","hostname":"Wait for a query_work_queue command to finish before sending IO
w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"downstairs","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138326628Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"":"downstairs","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.1383397Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138351802Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"0","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138364744Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"1","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138377206Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"client":"2","":"downstairs","session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138389809Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138401591Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"c8e5ffb5-4685-4079-b982-d2ac5a0e4515 is now active with session: f431f3f6-99da-4751-bf3a-db6f963333d8","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138413954Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
,"time":"2025-01-27T19:37:19.138425467Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302,"session_id":"f431f3f6-99da-4751-bf3a-db6f963333d8"}
{"msg":"The guest has finished waiting for activation with:30","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:19.138439308Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 505 len 8192 data: 3 4
02/30 Write block 271 len 12288 data: 4 4 4
03/30 Write block 161 len 24576 data: 3 2 2 4 4 4
04/30 Read block 175 len 20480
05/30 Write block 418 len 8192 data: 3 3
06/30 Write block 495 len 20480 data: 4 5 4 4 4
07/30 Read block 425 len 20480
08/30 Flush
09/30 Write block 554 len 32768 data: 3 4 4 5 5 7 6 4
10/30 Read block 244 len 28672
11/30 Read block 481 len 24576
12/30 Read block 525 len 28672
13/30 Read block 332 len 16384
14/30 Write block 177 len 24576 data: 5 5 6 6 7 6
15/30 Read block 477 len 4096
16/30 Read block 51 len 36864
17/30 Write block 3 len 16384 data: 3 3 4 3
18/30 Read block 342 len 24576
19/30 Read block 246 len 20480
20/30 Read block 550 len 20480
21/30 Read block 433 len 16384
22/30 Read block 90 len 40960
23/30 Write block 490 len 36864 data: 3 3 5 4 3 5 6 5 5
24/30 Flush
25/30 Write block 3 len 20480 data: 4 4 5 4 4
26/30 Write block 307 len 20480 data: 5 5 5 4 3
27/30 Read block 40 len 36864
28/30 Read block 314 len 8192
29/30 Write block 104 len 24576 data: 3 4 4 3 3 4
30/30 Write block 156 len 12288 data: 3 2 2
----------------------------------------------------------------
Crucible gen:30 GIO:true work queues: Upstairs:0 downstairs:6
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1024 Acked Write 5 Done Done Done false
1025 Acked Write 5 Done Done Done false
1026 Acked Read 9 Done Done Done false
1027 Acked Read 2 Done Done Done false
1028 Acked Write 6 Sent Sent Sent false
1029 Acked Write 3 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 4 4 4 12
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1023 1023 1023
Downstairs last five completed: 1023 1022 1021 1020 1019
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:19.158910831Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1302}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 30 30 30 53 53 53 T T T
1 020-039 29 29 29 51 51 51 F F F
2 040-059 28 28 28 47 47 47 F F F
3 060-079 28 28 28 49 49 49 F F F
4 080-099 23 23 23 39 39 39 F F F
5 100-119 20 20 20 32 32 32 T T T
6 120-139 22 22 22 37 37 37 F F F
7 140-159 23 23 23 39 39 39 T T T
8 160-179 30 30 30 53 53 53 F F F
9 180-199 30 30 30 53 53 53 F F F
10 200-219 28 28 28 48 48 48 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 30 30 30 52 52 52 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 26 26 26 45 45 45 T T T
16 320-339 29 29 29 51 51 51 F F F
17 340-359 29 29 29 51 51 51 F F F
18 360-379 24 24 24 42 42 42 F F F
19 380-399 23 23 23 39 39 39 F F F
20 400-419 30 30 30 52 52 52 F F F
21 420-439 26 26 26 45 45 45 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 25 25 25 44 44 44 F F F
24 480-499 30 30 30 53 53 53 F F F
25 500-519 30 30 30 52 52 52 F F F
26 520-539 24 24 24 42 42 42 F F F
27 540-559 30 30 30 53 53 53 F F F
28 560-579 30 30 30 53 53 53 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 30, Max flush: 53
A Difference in extent metadata is expected here
On loop 15
Verifying data now
No change after verify
Loop: 15 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 31 31 31 54 54 54 F F F
1 020-039 29 29 29 51 51 51 F F F
2 040-059 28 28 28 47 47 47 F F F
3 060-079 28 28 28 49 49 49 F F F
4 080-099 23 23 23 39 39 39 F F F
5 100-119 31 31 31 54 54 54 F F F
6 120-139 22 22 22 37 37 37 F F F
7 140-159 31 31 31 54 54 54 F F F
8 160-179 30 30 30 53 53 53 F F F
9 180-199 30 30 30 53 53 53 F F F
10 200-219 28 28 28 48 48 48 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 30 30 30 52 52 52 F F F
14 280-299 14 14 14 23 23 23 F F F
15 300-319 31 31 31 54 54 54 F F F
16 320-339 29 29 29 51 51 51 F F F
17 340-359 29 29 29 51 51 51 F F F
18 360-379 24 24 24 42 42 42 F F F
19 380-399 23 23 23 39 39 39 F F F
20 400-419 30 30 30 52 52 52 F F F
21 420-439 26 26 26 45 45 45 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 25 25 25 44 44 44 F F F
24 480-499 30 30 30 53 53 53 F F F
25 500-519 30 30 30 52 52 52 F F F
26 520-539 24 24 24 42 42 42 F F F
27 540-559 30 30 30 53 53 53 F F F
28 560-579 30 30 30 53 53 53 F F F
29 580-599 11 11 11 17 17 17 F F F
Max gen: 31, Max flush: 54
Begin loop 16
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.443296903Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.489140495Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.490301016Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.4903443Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.49035969Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.490378239Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310}
{"msg":"Crucible 0ec1b82b-8a64-4e98-8cbb-10e9f6296070 has session id: 5f7ad370-8ed3-467d-b6fe-065b2522f0da","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.491029505Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"Upstairs opts: Upstairs UUID: 0ec1b82b-8a64-4e98-8cbb-10e9f6296070, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.491075627Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"Crucible stats registered with UUID: 0ec1b82b-8a64-4e98-8cbb-10e9f6296070","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.49110261Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:23.491121259Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.491678643Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"task":"crutest"}
{"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.491705077Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310}
{"msg":"0ec1b82b-8a64-4e98-8cbb-10e9f6296070 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.491772645Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.491790254Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"0","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.491815888Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"0","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.491830429Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"1","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.49184504Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"1","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.491858852Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"2","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.491873543Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"2","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.491887024Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"io task","client":"2","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.491901046Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"io task","client":"0","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.491915097Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"io task","client":"1","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"ds_connection connected from Ok(127.0.0.1:40523)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.49203972Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"io task","client":"0","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:23.492092777Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"io task","client":"2","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:23.492134461Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"io task","client":"2","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:23.492167451Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"ds_connection connected from Ok(127.0.0.1:32819)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.492419675Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"io task","client":"1","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.49245818Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"io task","client":"2","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.492972602Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"0","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.493001044Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"1","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.493603391Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"1","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.493731002Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"0","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.494178455Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:23.494280312Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.493324817Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"io task","client":"2","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"ds_connection connected from Ok(127.0.0.1:35840)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.493425264Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"io task","client":"2","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.49462718Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"2","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.495263526Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"2","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"[0]R flush_numbers[0..12]: [54, 51, 47, 49, 39, 54, 37, 54, 53, 53, 48, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.49595234Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"downstairs","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"[0]R generation[0..12]: [31, 29, 28, 28, 23, 31, 22, 31, 30, 30, 28, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.495980992Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"downstairs","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.49599971Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"downstairs","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"[1]R flush_numbers[0..12]: [54, 51, 47, 49, 39, 54, 37, 54, 53, 53, 48, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.4960168Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"downstairs","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"[1]R generation[0..12]: [31, 29, 28, 28, 23, 31, 22, 31, 30, 30, 28, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496033349Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"downstairs","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496050439Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"downstairs","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"[2]R flush_numbers[0..12]: [54, 51, 47, 49, 39, 54, 37, 54, 53, 53, 48, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496067059Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"downstairs","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"[2]R generation[0..12]: [31, 29, 28, 28, 23, 31, 22, 31, 30, 30, 28, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496083449Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"downstairs","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496100178Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"downstairs","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496115529Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"downstairs","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"Generation requested: 32 >= found:32","v":0Wait for a query_work_queue command to finish before sending IO
,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496138834Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"downstairs","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"Next flush: 55","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496154434Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"downstairs","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496169805Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"":"downstairs","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496186245Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496205193Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"0","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496221803Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"1","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496237873Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"client":"2","":"downstairs","session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496254083Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
,"time":"2025-01-27T19:37:33.496269034Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"0ec1b82b-8a64-4e98-8cbb-10e9f6296070 is now active with session: 5f7ad370-8ed3-467d-b6fe-065b2522f0da","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496287362Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496301983Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310,"session_id":"5f7ad370-8ed3-467d-b6fe-065b2522f0da"}
{"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:33.496316904Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 521 len 12288 data: 5 6 6
02/30 Write block 213 len 24576 data: 4 4 4 2 2 2
03/30 Write block 206 len 4096 data: 5
04/30 Write block 327 len 4096 data: 7
05/30 Write block 133 len 8192 data: 3 4
06/30 Read block 6 len 24576
07/30 Write block 291 len 8192 data: 3 3
08/30 Write block 377 len 36864 data: 5 5 5 4 2 2 3 3 3
09/30 Read block 544 len 16384
10/30 Read block 431 len 24576
11/30 Read block 31 len 32768
12/30 Read block 105 len 8192
13/30 Read block 127 len 4096
14/30 Write block 330 len 20480 data: 3 2 3 3 2
15/30 Read block 91 len 12288
16/30 Read block 103 len 40960
17/30 Write block 464 len 32768 data: 3 2 2 2 2 3 3 2
18/30 Read block 588 len 36864
19/30 Write block 551 len 8192 data: 3 3
20/30 Write block 264 len 36864 data: 2 2 2 2 3 4 4 5 5
21/30 Read block 279 len 12288
22/30 Flush
23/30 Write block 402 len 28672 data: 3 3 3 3 4 4 4
24/30 Write block 487 len 40960 data: 4 4 3 4 4 6 5 4 6 7
25/30 Write block 18 len 36864 data: 3 3 3 3 4 6 6 5 6
26/30 Write block 495 len 20480 data: 7 8 6 6 5
27/30 Read block 319 len 24576
28/30 Write block 589 len 40960 data: 2 2 2 2 2 2 2 2 2 2
29/30 Write block 92 len 40960 data: 5 4 4 4 3 2 2 2 3 3
30/30 Write block 429 len 8192 data: 4 3
----------------------------------------------------------------
Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:8
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1022 Acked Write 7 Done Done Done false
1023 Acked Write 10 Done Done Done false
1024 Acked Write 9 Done Done Done false
1025 Acked Write 5 Done Done Done false
1026 Acked Read 6 Done Done Done false
1027 Acked Write 10 Done Sent Done false
1028 Acked Write 10 Sent Sent Sent false
1029 Acked Write 2 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 3 2 7
Done 6 5 6 17
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1021 1021 1021
Downstairs last five completed: 1021 1020 1019 1018 1017
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:33.515419615Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1310}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 31 31 31 54 54 54 T T T
1 020-039 29 29 29 51 51 51 T T T
2 040-059 28 28 28 47 47 47 F F F
3 060-079 28 28 28 49 49 49 F F F
4 080-099 23 23 23 39 39 39 T T T
5 100-119 31 31 31 54 54 54 T T T
6 120-139 32 32 32 55 55 55 F F F
7 140-159 31 31 31 54 54 54 F F F
8 160-179 30 30 30 53 53 53 F F F
9 180-199 30 30 30 53 53 53 F F F
10 200-219 32 32 32 55 55 55 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 32 32 32 55 55 55 F F F
14 280-299 32 32 32 55 55 55 F F F
15 300-319 31 31 31 54 54 54 F F F
16 320-339 32 32 32 55 55 55 F F F
17 340-359 29 29 29 51 51 51 F F F
18 360-379 32 32 32 55 55 55 F F F
19 380-399 32 32 32 55 55 55 F F F
20 400-419 30 30 30 52 52 52 T T T
21 420-439 26 26 26 45 45 45 T T T
22 440-459 26 26 26 45 45 45 F F F
23 460-479 32 32 32 55 55 55 F F F
24 480-499 30 30 30 53 53 53 T T T
25 500-519 30 30 30 52 52 52 F F F
26 520-539 32 32 32 55 55 55 F F F
27 540-559 32 32 32 55 55 55 F F F
28 560-579 30 30 30 53 53 53 F F F
29 580-599 11 11 11 17 17 17 T T T
Max gen: 32, Max flush: 55
A Difference in extent metadata is expected here
On loop 16
Verifying data now
No change after verify
Loop: 16 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 33 33 33 56 56 56 F F F
1 020-039 33 33 33 56 56 56 F F F
2 040-059 28 28 28 47 47 47 F F F
3 060-079 28 28 28 49 49 49 F F F
4 080-099 33 33 33 56 56 56 F F F
5 100-119 33 33 33 56 56 56 F F F
6 120-139 32 32 32 55 55 55 F F F
7 140-159 31 31 31 54 54 54 F F F
8 160-179 30 30 30 53 53 53 F F F
9 180-199 30 30 30 53 53 53 F F F
10 200-219 32 32 32 55 55 55 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 32 32 32 55 55 55 F F F
14 280-299 32 32 32 55 55 55 F F F
15 300-319 31 31 31 54 54 54 F F F
16 320-339 32 32 32 55 55 55 F F F
17 340-359 29 29 29 51 51 51 F F F
18 360-379 32 32 32 55 55 55 F F F
19 380-399 32 32 32 55 55 55 F F F
20 400-419 33 33 33 56 56 56 F F F
21 420-439 33 33 33 56 56 56 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 32 32 32 55 55 55 F F F
24 480-499 33 33 33 56 56 56 F F F
25 500-519 30 30 30 52 52 52 F F F
26 520-539 32 32 32 55 55 55 F F F
27 540-559 32 32 32 55 55 55 F F F
28 560-579 30 30 30 53 53 53 F F F
29 580-599 33 33 33 56 56 56 F F F
Max gen: 33, Max flush: 56
Begin loop 17
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.791867621Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:37.835364986Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.836014313Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.868099305Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.869024452Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.869063198Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.869077869Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.86909222Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318}
{"msg":"Crucible 2cf6ebee-50a5-4128-aca8-6392ae7de979 has session id: 7c7998d5-b922-442f-bcc1-9eb1e0bb93fa","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.869694797Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"Upstairs opts: Upstairs UUID: 2cf6ebee-50a5-4128-aca8-6392ae7de979, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.869736561Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"Crucible stats registered with UUID: 2cf6ebee-50a5-4128-aca8-6392ae7de979","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.869752721Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:37.869765324Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.870196067Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"task":"crutest"}
{"msg":"The guest has requested activation with gen:34","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.870229816Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318}
{"msg":"2cf6ebee-50a5-4128-aca8-6392ae7de979 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.870289309Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.870315643Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"0","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.870392185Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"0","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.870434039Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"1","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.870460643Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"1","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.87048224Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"2","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.870507434Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"2","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.87053029Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"io task","client":"2","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.870556474Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"io task","client":"0","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.87057871Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"io task","client":"1","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"ds_connection connected from Ok(127.0.0.1:51672)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.870601905Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"io task","client":"2","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"ds_connection connected from Ok(127.0.0.1:61592)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.87062588Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"io task","client":"0","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"ds_connection connected from Ok(127.0.0.1:54286)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.870650025Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"io task","client":"1","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.871345905Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"2","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.87138533Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"1","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.871501618Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"0","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.87201577Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"2","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872056685Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"1","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872173352Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"0","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872605125Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872690152Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"[0]R flush_numbers[0..12]: [56, 56, 47, 49, 56, 56, 55, 54, 53, 53, 55, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872808649Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"downstairs","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"[0]R generation[0..12]: [33, 33, 28, 28, 33, 33, 32, 31, 30, 30, 32, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872827957Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"downstairs","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872842338Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"downstairs","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"[1]R flush_numbers[0..12]: [56, 56, 47, 49, 56, 56, 55, 54, 53, 53, 55, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872856229Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"downstairs","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"[1]R generation[0..12]: [33, 33, 28, 28, 33, 33, 32, 31, 30, 30, 32, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872869761Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"downstairs","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872883453Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"downstairs","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"[2]R flush_numbers[0..12]: [56, 56, 47, 49, 56, 56, 55, 54, 53, 53, 55, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872896614Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"downstairs","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"[2]R generation[0..12]: [33, 33, 28, 28, 33, 33, 32, 31, 30, 30, 32, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872909876Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"downstairs","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872928445Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"downstairs","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"Max found gen is 34","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872941257Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"downstairs","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"Generation requested: 34 >= found:34","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872953909Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"downstairs","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"Next flush: 57","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872966371Z","Wait for a query_work_queue command to finish before sending IO
hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"downstairs","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.872993325Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"":"downstairs","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.873018859Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.873043094Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"0","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.873067199Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"1","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.873092573Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"client":"2","":"downstairs","session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.873118407Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.873142732Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"2cf6ebee-50a5-4128-aca8-6392ae7de979 is now active with session: 7c7998d5-b922-442f-bcc1-9eb1e0bb93fa","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.873168976Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:37.873193131Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318,"session_id":"7c7998d5-b922-442f-bcc1-9eb1e0bb93fa"}
{"msg":"The guest has finished waiting for activation with:34","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
,"time":"2025-01-27T19:37:37.873218355Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 519 len 12288 data: 3 4 6
02/30 Read block 367 len 4096
03/30 Write block 84 len 4096 data: 4
04/30 Write block 524 len 40960 data: 6 5 5 8 9 8 8 8 7 7
05/30 Read block 137 len 16384
06/30 Write block 149 len 32768 data: 5 4 4 4 7 6 3 4
07/30 Read block 247 len 4096
08/30 Write block 96 len 40960 data: 4 3 3 3 4 4 4 3 4 5
09/30 Read block 244 len 24576
10/30 Read block 77 len 28672
11/30 Read block 455 len 16384
12/30 Flush
13/30 Write block 103 len 40960 data: 4 5 6 5 4 4 5 4 4 4
14/30 Flush
15/30 Write block 56 len 28672 data: 3 3 3 2 2 2 3
16/30 Read block 134 len 12288
17/30 Write block 488 len 8192 data: 5 4
18/30 Write block 540 len 32768 data: 2 3 2 2 2 2 2 2
19/30 Read block 37 len 40960
20/30 Write block 80 len 24576 data: 4 4 4 4 5 3
21/30 Write block 117 len 24576 data: 5 5 5 4 4 3
22/30 Read block 78 len 20480
23/30 Read block 207 len 40960
24/30 Read block 235 len 12288
25/30 Read block 474 len 24576
26/30 Read block 123 len 40960
27/30 Write block 5 len 12288 data: 6 5 5
28/30 Read block 68 len 12288
29/30 Write block 204 len 24576 data: 6 7 6 6 6 5
30/30 Write block 90 len 16384 data: 5 4 6 5
----------------------------------------------------------------
Crucible gen:34 GIO:true work queues: Upstairs:0 downstairs:16
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1014 Acked Write 7 Done Done Done false
1015 Acked Read 3 Done Done Done false
1016 Acked Write 2 Done Done Done false
1017 Acked Write 8 Done Done Done false
1018 Acked Read 10 Done Done Done false
1019 Acked Write 6 Done Done Done false
1020 Acked Write 6 Done Done Done false
1021 Acked Read 5 Done Done Done false
1022 Acked Read 10 Done Done Done false
1023 Acked Read 3 Done Done Done false
1024 Acked Read 6 Done Done Done false
1025 Acked Read 10 Done Done Done false
1026 Acked Write 3 Done Done Done false
1027 Acked Read 3 Done Done Done false
1028 Acked Write 6 Sent Sent Sent false
1029 Acked Write 4 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 14 14 14 42
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1013 1013 1013
Downstairs last five completed: 1013 1012 1011 1010 1009
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:37.89509239Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1318}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 33 33 33 56 56 56 T T T
1 020-039 33 33 33 56 56 56 F F F
2 040-059 28 28 28 47 47 47 T T T
3 060-079 28 28 28 49 49 49 T T T
4 080-099 34 34 34 57 57 57 T T T
5 100-119 34 34 34 58 58 58 T T T
6 120-139 32 32 32 55 55 55 T T T
7 140-159 34 34 34 57 57 57 F F F
8 160-179 30 30 30 53 53 53 F F F
9 180-199 30 30 30 53 53 53 F F F
10 200-219 32 32 32 55 55 55 T T T
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 32 32 32 55 55 55 F F F
14 280-299 32 32 32 55 55 55 F F F
15 300-319 31 31 31 54 54 54 F F F
16 320-339 32 32 32 55 55 55 F F F
17 340-359 29 29 29 51 51 51 F F F
18 360-379 32 32 32 55 55 55 F F F
19 380-399 32 32 32 55 55 55 F F F
20 400-419 33 33 33 56 56 56 F F F
21 420-439 33 33 33 56 56 56 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 32 32 32 55 55 55 F F F
24 480-499 33 33 33 56 56 56 T T T
25 500-519 34 34 34 57 57 57 F F F
26 520-539 34 34 34 57 57 57 F F F
27 540-559 32 32 32 55 55 55 T T T
28 560-579 30 30 30 53 53 53 F F F
29 580-599 33 33 33 56 56 56 F F F
Max gen: 34, Max flush: 58
A Difference in extent metadata is expected here
On loop 17
Verifying data now
No change after verify
Loop: 17 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 35 35 35 59 59 59 F F F
1 020-039 33 33 33 56 56 56 F F F
2 040-059 35 35 35 59 59 59 F F F
3 060-079 35 35 35 59 59 59 F F F
4 080-099 35 35 35 59 59 59 F F F
5 100-119 35 35 35 59 59 59 F F F
6 120-139 35 35 35 59 59 59 F F F
7 140-159 34 34 34 57 57 57 F F F
8 160-179 30 30 30 53 53 53 F F F
9 180-199 30 30 30 53 53 53 F F F
10 200-219 35 35 35 59 59 59 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 32 32 32 55 55 55 F F F
14 280-299 32 32 32 55 55 55 F F F
15 300-319 31 31 31 54 54 54 F F F
16 320-339 32 32 32 55 55 55 F F F
17 340-359 29 29 29 51 51 51 F F F
18 360-379 32 32 32 55 55 55 F F F
19 380-399 32 32 32 55 55 55 F F F
20 400-419 33 33 33 56 56 56 F F F
21 420-439 33 33 33 56 56 56 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 32 32 32 55 55 55 F F F
24 480-499 35 35 35 59 59 59 F F F
25 500-519 34 34 34 57 57 57 F F F
26 520-539 34 34 34 57 57 57 F F F
27 540-559 35 35 35 59 59 59 F F F
28 560-579 30 30 30 53 53 53 F F F
29 580-599 33 33 33 56 56 56 F F F
Max gen: 35, Max flush: 59
Begin loop 18
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.206178699Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:42.250064783Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.250773613Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.282733473Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.283703592Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.283746775Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.283769331Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.28378748Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326}
{"msg":"Crucible 5b82ccce-510d-4d24-be4f-e5c60942bb13 has session id: f9fa2d62-4354-414d-bd53-4baf7f991013","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.284424615Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"Upstairs opts: Upstairs UUID: 5b82ccce-510d-4d24-be4f-e5c60942bb13, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.284478172Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"Crucible stats registered with UUID: 5b82ccce-510d-4d24-be4f-e5c60942bb13","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.28449773Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:42.284510732Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.284855259Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"task":"crutest"}
{"msg":"The guest has requested activation with gen:36","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.284888858Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326}
{"msg":"5b82ccce-510d-4d24-be4f-e5c60942bb13 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.284945273Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.284972376Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"0","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.284999939Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"0","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.28501453Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"1","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.285028881Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"1","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.285041903Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"2","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.285055984Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"2","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.285094211Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"io task","client":"2","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.285117106Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"io task","client":"0","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.285137404Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"io task","client":"1","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52062)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.285210728Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"io task","client":"2","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39547)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.28551601Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"io task","client":"1","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"ds_connection connected from Ok(127.0.0.1:38334)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.285550998Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"io task","client":"0","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.286024035Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"1","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.286066349Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"2","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.286318502Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"0","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.286829017Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"1","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.286857979Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"2","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.286873069Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"0","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287385282Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287410846Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"[0]R flush_numbers[0..12]: [59, 56, 59, 59, 59, 59, 59, 57, 53, 53, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287477265Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"downstairs","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"[0]R generation[0..12]: [35, 33, 35, 35, 35, 35, 35, 34, 30, 30, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287500411Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"downstairs","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287515621Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"downstairs","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"[1]R flush_numbers[0..12]: [59, 56, 59, 59, 59, 59, 59, 57, 53, 53, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287529283Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"downstairs","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"[1]R generation[0..12]: [35, 33, 35, 35, 35, 35, 35, 34, 30, 30, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287542644Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"downstairs","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287556196Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"downstairs","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"[2]R flush_numbers[0..12]: [59, 56, 59, 59, 59, 59, 59, 57, 53, 53, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287569518Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"downstairs","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"[2]R generation[0..12]: [35, 33, 35, 35, 35, 35, 35, 34, 30, 30, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.28758266Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"downstairs","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287596381Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"downstairs","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"Max found gen is 36","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287608923Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"downstairs","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"Generation requested: 36 >= found:36","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287621746Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"downstairs","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"Next flush: 60","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287634248Z",Wait for a query_work_queue command to finish before sending IO
"hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"downstairs","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287654066Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"":"downstairs","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287666947Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.28767863Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"0","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287693281Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"1","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287706183Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"client":"2","":"downstairs","session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287719095Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287730938Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"5b82ccce-510d-4d24-be4f-e5c60942bb13 is now active with session: f9fa2d62-4354-414d-bd53-4baf7f991013","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.28774342Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287755113Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326,"session_id":"f9fa2d62-4354-414d-bd53-4baf7f991013"}
{"msg":"The guest has finished waiting for activation with:36","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:42.287767006Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 263 len 16384 data: 3 3 3 3
02/30 Read block 326 len 32768
03/30 Read block 37 len 8192
04/30 Read block 507 len 28672
05/30 Write block 530 len 28672 data: 9 9 8 8 7 7 5
06/30 Write block 537 len 12288 data: 4 3 2
07/30 Read block 181 len 40960
08/30 Write block 191 len 4096 data: 3
09/30 Write block 357 len 20480 data: 5 6 5 5 3
10/30 Write block 547 len 12288 data: 3 2 2
11/30 Read block 285 len 32768
12/30 Flush
13/30 Write block 338 len 12288 data: 4 5 5
14/30 Write block 293 len 20480 data: 3 2 2 2 4
15/30 Read block 226 len 16384
16/30 Read block 543 len 36864
17/30 Read block 189 len 36864
18/30 Write block 3 len 36864 data: 5 5 7 6 6 4 4 4 3
19/30 Write block 6 len 24576 data: 7 7 5 5 5 4
20/30 Read block 110 len 16384
21/30 Flush
22/30 Write block 296 len 32768 data: 3 5 3 4 5 5 3 5
23/30 Write block 49 len 28672 data: 2 2 3 4 4 4 4
24/30 Read block 171 len 36864
25/30 Write block 292 len 12288 data: 4 4 3
26/30 Write block 354 len 28672 data: 5 5 6 6 7 6 6
27/30 Write block 378 len 20480 data: 6 6 5 3 3
28/30 Write block 367 len 40960 data: 2 2 4 4 4 4 4 4 4 4
29/30 Read block 26 len 24576
30/30 Write block 46 len 36864 data: 3 3 2 3 3 4 5 5 5
----------------------------------------------------------------
Crucible gen:36 GIO:true work queues: Upstairs:0 downstairs:9
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1021 Acked Write 8 Done Done Done false
1022 Acked Write 7 Done Done Done false
1023 Acked Read 9 Done Done Done false
1024 Acked Write 3 Done Done Done false
1025 Acked Write 7 Done Done Done false
1026 Acked Write 5 Done Done Done false
1027 Acked Write 10 Done Done Done false
1028 Acked Read 6 Done Done Done false
1029 Acked Write 9 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 8 8 8 24
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1020 1020 1020
Downstairs last five completed: 1020 1019 1018 1017 1016
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:42.307795103Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1326}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 36 36 36 61 61 61 F F F
1 020-039 33 33 33 56 56 56 F F F
2 040-059 35 35 35 59 59 59 T T T
3 060-079 35 35 35 59 59 59 F F F
4 080-099 35 35 35 59 59 59 F F F
5 100-119 35 35 35 59 59 59 F F F
6 120-139 35 35 35 59 59 59 F F F
7 140-159 34 34 34 57 57 57 F F F
8 160-179 30 30 30 53 53 53 F F F
9 180-199 36 36 36 60 60 60 F F F
10 200-219 35 35 35 59 59 59 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 36 36 36 60 60 60 F F F
14 280-299 36 36 36 61 61 61 T T T
15 300-319 31 31 31 54 54 54 T T T
16 320-339 36 36 36 61 61 61 F F F
17 340-359 36 36 36 61 61 61 T T T
18 360-379 36 36 36 60 60 60 T T T
19 380-399 32 32 32 55 55 55 T T T
20 400-419 33 33 33 56 56 56 F F F
21 420-439 33 33 33 56 56 56 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 32 32 32 55 55 55 F F F
24 480-499 35 35 35 59 59 59 F F F
25 500-519 34 34 34 57 57 57 F F F
26 520-539 36 36 36 60 60 60 F F F
27 540-559 36 36 36 60 60 60 F F F
28 560-579 30 30 30 53 53 53 F F F
29 580-599 33 33 33 56 56 56 F F F
Max gen: 36, Max flush: 61
A Difference in extent metadata is expected here
On loop 18
Verifying data now
No change after verify
Loop: 18 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 36 36 36 61 61 61 F F F
1 020-039 33 33 33 56 56 56 F F F
2 040-059 37 37 37 62 62 62 F F F
3 060-079 35 35 35 59 59 59 F F F
4 080-099 35 35 35 59 59 59 F F F
5 100-119 35 35 35 59 59 59 F F F
6 120-139 35 35 35 59 59 59 F F F
7 140-159 34 34 34 57 57 57 F F F
8 160-179 30 30 30 53 53 53 F F F
9 180-199 36 36 36 60 60 60 F F F
10 200-219 35 35 35 59 59 59 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 36 36 36 60 60 60 F F F
14 280-299 37 37 37 62 62 62 F F F
15 300-319 37 37 37 62 62 62 F F F
16 320-339 36 36 36 61 61 61 F F F
17 340-359 37 37 37 62 62 62 F F F
18 360-379 37 37 37 62 62 62 F F F
19 380-399 37 37 37 62 62 62 F F F
20 400-419 33 33 33 56 56 56 F F F
21 420-439 33 33 33 56 56 56 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 32 32 32 55 55 55 F F F
24 480-499 35 35 35 59 59 59 F F F
25 500-519 34 34 34 57 57 57 F F F
26 520-539 36 36 36 60 60 60 F F F
27 540-559 36 36 36 60 60 60 F F F
28 560-579 30 30 30 53 53 53 F F F
29 580-599 33 33 33 56 56 56 F F F
Max gen: 37, Max flush: 62
Begin loop 19
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.595615619Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:46.640186868Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.640818397Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.673476004Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.67456445Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.674614549Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.674631489Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.674649477Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334}
{"msg":"Crucible 64aa0926-fe1b-4b9f-a2f5-d4dd9566ec87 has session id: 0bfaf042-8f78-47e5-890e-0a528e8a61ca","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.675303142Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"Upstairs opts: Upstairs UUID: 64aa0926-fe1b-4b9f-a2f5-d4dd9566ec87, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.675346336Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"Crucible stats registered with UUID: 64aa0926-fe1b-4b9f-a2f5-d4dd9566ec87","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.675373259Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:46.675393506Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.675778288Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"task":"crutest"}
{"msg":"The guest has requested activation with gen:38","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.675810478Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334}
{"msg":"64aa0926-fe1b-4b9f-a2f5-d4dd9566ec87 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.675867253Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.675903271Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"0","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.675934302Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"0","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.675949222Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"1","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.675964103Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"1","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.675977445Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"2","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.675991436Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"2","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.676004678Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"io task","client":"2","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.67601827Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"io task","client":"0","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.676035439Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"io task","client":"1","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39541)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.676096701Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"io task","client":"2","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"ds_connection connected from Ok(127.0.0.1:60765)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.676125703Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"io task","client":"0","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50273)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.676149918Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"io task","client":"1","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.676859439Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"1","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.676901713Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"2","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.677161642Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"0","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.677627323Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"1","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.677654497Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"2","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.677718437Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"0","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678298867Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678323122Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"[0]R flush_numbers[0..12]: [61, 56, 62, 59, 59, 59, 59, 57, 53, 60, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678393719Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"downstairs","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"[0]R generation[0..12]: [36, 33, 37, 35, 35, 35, 35, 34, 30, 36, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678413267Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"downstairs","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678427658Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"downstairs","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"[1]R flush_numbers[0..12]: [61, 56, 62, 59, 59, 59, 59, 57, 53, 60, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678441309Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"downstairs","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"[1]R generation[0..12]: [36, 33, 37, 35, 35, 35, 35, 34, 30, 36, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678454581Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"downstairs","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678468282Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"downstairs","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"[2]R flush_numbers[0..12]: [61, 56, 62, 59, 59, 59, 59, 57, 53, 60, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678481494Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"downstairs","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"[2]R generation[0..12]: [36, 33, 37, 35, 35, 35, 35, 34, 30, 36, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678494676Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"downstairs","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678512615Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"downstairs","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"Max found gen is 38","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678525507Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"downstairs","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"Generation requested: 38 >= found:38","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678538339Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"downstairs","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"Next flush: 63","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678550662Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"downstairs","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO
2025-01-27T19:37:46.678563084Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"":"downstairs","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678583751Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678595544Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"0","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678608466Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"1","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678620918Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"client":"2","":"downstairs","session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.67863352Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678645503Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"64aa0926-fe1b-4b9f-a2f5-d4dd9566ec87 is now active with session: 0bfaf042-8f78-47e5-890e-0a528e8a61ca","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678657955Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678669538Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334,"session_id":"0bfaf042-8f78-47e5-890e-0a528e8a61ca"}
{"msg":"The guest has finished waiting for activation with:38","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:46.678681321Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 175 len 32768 data: 5 5 6 6 7 7 8 7
02/30 Flush
03/30 Write block 88 len 28672 data: 3 3 6 5 7 6 5
04/30 Flush
05/30 Write block 55 len 20480 data: 5 4 4 4 3
06/30 Write block 187 len 36864 data: 4 2 2 2 4 3 3 4 7
07/30 Read block 207 len 36864
08/30 Write block 400 len 28672 data: 3 3 4 4 4 4 5
09/30 Read block 193 len 12288
10/30 Write block 81 len 28672 data: 5 5 5 6 4 4 3
11/30 Write block 486 len 8192 data: 4 5
12/30 Read block 126 len 36864
13/30 Read block 431 len 40960
14/30 Read block 123 len 28672
15/30 Read block 262 len 16384
16/30 Read block 31 len 32768
17/30 Write block 145 len 36864 data: 5 6 6 6 6 5 5 5 8
18/30 Write block 572 len 20480 data: 5 5 5 4 4
19/30 Read block 460 len 40960
20/30 Read block 284 len 40960
21/30 Write block 311 len 32768 data: 4 3 3 4 3 3 3 3
22/30 Write block 113 len 20480 data: 2 2 3 5 6
23/30 Write block 279 len 8192 data: 5 4
24/30 Write block 131 len 8192 data: 2 2
25/30 Read block 304 len 16384
26/30 Write block 171 len 24576 data: 5 5 3 4 6 6
27/30 Write block 307 len 12288 data: 6 6 6
28/30 Write block 585 len 24576 data: 3 3 2 2 3 3
29/30 Read block 379 len 8192
30/30 Read block 566 len 20480
----------------------------------------------------------------
Crucible gen:38 GIO:true work queues: Upstairs:0 downstairs:26
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1004 Acked Write 5 Done Done Done false
1005 Acked Write 9 Done Done Done false
1006 Acked Read 9 Done Done Done false
1007 Acked Write 7 Done Done Done false
1008 Acked Read 3 Done Done Done false
1009 Acked Write 7 Done Done Done false
1010 Acked Write 2 Done Done Done false
1011 Acked Read 9 Done Done Done false
1012 Acked Read 10 Done Done Done false
1013 Acked Read 7 Done Done Done false
1014 Acked Read 4 Done Done Done false
1015 Acked Read 8 Done Done Done false
1016 Acked Write 9 Done Done Done false
1017 Acked Write 5 Done Done Done false
1018 Acked Read 10 Done Done Done false
1019 Acked Read 10 Done Done Done false
1020 Acked Write 8 Done Done Done false
1021 Acked Write 5 Done Done Done false
1022 Acked Write 2 Done Done Done false
1023 Acked Write 2 Done Done Done false
1024 Acked Read 4 Done Done Done false
1025 Acked Write 6 Done Done Done false
1026 Acked Write 3 Done Done Done false
1027 Acked Write 6 Done Done Done false
1028 Acked Read 2 Done Done Done false
1029 Acked Read 5 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 0 1
Done 26 25 26 77
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1003 1003 1003
Downstairs last five completed: 1003 1002 1001 1000
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:46.699512561Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1334}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 36 36 36 61 61 61 F F F
1 020-039 33 33 33 56 56 56 F F F
2 040-059 37 37 37 62 62 62 T T T
3 060-079 35 35 35 59 59 59 F F F
4 080-099 38 38 38 64 64 64 T T T
5 100-119 35 35 35 59 59 59 T T T
6 120-139 35 35 35 59 59 59 T T T
7 140-159 34 34 34 57 57 57 T T T
8 160-179 38 38 38 63 63 63 T T T
9 180-199 38 38 38 63 63 63 T T T
10 200-219 35 35 35 59 59 59 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 36 36 36 60 60 60 T T T
14 280-299 37 37 37 62 62 62 T T T
15 300-319 37 37 37 62 62 62 T T T
16 320-339 36 36 36 61 61 61 F F F
17 340-359 37 37 37 62 62 62 F F F
18 360-379 37 37 37 62 62 62 F F F
19 380-399 37 37 37 62 62 62 F F F
20 400-419 33 33 33 56 56 56 T T T
21 420-439 33 33 33 56 56 56 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 32 32 32 55 55 55 F F F
24 480-499 35 35 35 59 59 59 T T T
25 500-519 34 34 34 57 57 57 F F F
26 520-539 36 36 36 60 60 60 F F F
27 540-559 36 36 36 60 60 60 F F F
28 560-579 30 30 30 53 53 53 T T T
29 580-599 33 33 33 56 56 56 T T T
Max gen: 38, Max flush: 64
A Difference in extent metadata is expected here
On loop 19
Verifying data now
No change after verify
Loop: 19 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 36 36 36 61 61 61 F F F
1 020-039 33 33 33 56 56 56 F F F
2 040-059 39 39 39 65 65 65 F F F
3 060-079 35 35 35 59 59 59 F F F
4 080-099 39 39 39 65 65 65 F F F
5 100-119 39 39 39 65 65 65 F F F
6 120-139 39 39 39 65 65 65 F F F
7 140-159 39 39 39 65 65 65 F F F
8 160-179 39 39 39 65 65 65 F F F
9 180-199 39 39 39 65 65 65 F F F
10 200-219 35 35 35 59 59 59 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 39 39 39 65 65 65 F F F
14 280-299 39 39 39 65 65 65 F F F
15 300-319 39 39 39 65 65 65 F F F
16 320-339 36 36 36 61 61 61 F F F
17 340-359 37 37 37 62 62 62 F F F
18 360-379 37 37 37 62 62 62 F F F
19 380-399 37 37 37 62 62 62 F F F
20 400-419 39 39 39 65 65 65 F F F
21 420-439 33 33 33 56 56 56 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 32 32 32 55 55 55 F F F
24 480-499 39 39 39 65 65 65 F F F
25 500-519 34 34 34 57 57 57 F F F
26 520-539 36 36 36 60 60 60 F F F
27 540-559 36 36 36 60 60 60 F F F
28 560-579 39 39 39 65 65 65 F F F
29 580-599 39 39 39 65 65 65 F F F
Max gen: 39, Max flush: 65
Begin loop 20
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.041101311Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.087397472Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.088419759Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.08846718Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.088488896Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.088504087Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342}
{"msg":"Crucible 42cc55a9-a20b-4311-a626-ac5d85a58d88 has session id: 98afb387-22b9-4baa-bfe2-a8c33ced2aee","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.089092173Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"Upstairs opts: Upstairs UUID: 42cc55a9-a20b-4311-a626-ac5d85a58d88, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.089150017Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"Crucible stats registered with UUID: 42cc55a9-a20b-4311-a626-ac5d85a58d88","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.08917732Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:51.089200485Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.089654344Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"task":"crutest"}
{"msg":"The guest has requested activation with gen:40","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.089688093Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342}
{"msg":"42cc55a9-a20b-4311-a626-ac5d85a58d88 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.089746457Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.089776599Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"0","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.08980596Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"0","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.089820871Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"1","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.089835682Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"1","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.089849124Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"2","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.089863045Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"2","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.089893756Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"io task","client":"2","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.089927475Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"io task","client":"0","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.089953449Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"io task","client":"1","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:51.089992275Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"io task","client":"2","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:51.090025535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"io task","client":"2","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:37:51.090054966Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52930)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.090076393Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"io task","client":"0","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50848)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.090100648Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"io task","client":"1","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.090325289Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"io task","client":"2","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.090710131Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"1","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.090782586Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"0","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.091332345Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"1","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.091481823Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"0","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.091872341Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:37:51.092037729Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.09110279Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"io task","client":"2","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"ds_connection connected from Ok(127.0.0.1:42469)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.091242763Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"io task","client":"2","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.092389973Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"2","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093009189Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"2","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"[0]R flush_numbers[0..12]: [61, 56, 65, 59, 65, 65, 65, 65, 65, 65, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093666652Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"downstairs","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"[0]R generation[0..12]: [36, 33, 39, 35, 39, 39, 39, 39, 39, 39, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093694755Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"downstairs","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093714083Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"downstairs","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"[1]R flush_numbers[0..12]: [61, 56, 65, 59, 65, 65, 65, 65, 65, 65, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093731742Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"downstairs","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"[1]R generation[0..12]: [36, 33, 39, 35, 39, 39, 39, 39, 39, 39, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093749371Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"downstairs","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.09376724Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"downstairs","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"[2]R flush_numbers[0..12]: [61, 56, 65, 59, 65, 65, 65, 65, 65, 65, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093784599Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"downstairs","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"[2]R generation[0..12]: [36, 33, 39, 35, 39, 39, 39, 39, 39, 39, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093801509Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"downstairs","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
Wait for a query_work_queue command to finish before sending IO
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093825984Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"downstairs","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"Max found gen is 40","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093850289Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"downstairs","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"Generation requested: 40 >= found:40","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093868277Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"downstairs","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"Next flush: 66","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093884337Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"downstairs","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093900088Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"":"downstairs","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093916158Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093931039Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"0","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093947418Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"1","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093963428Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"client":"2","":"downstairs","session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093979818Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.093994909Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"42cc55a9-a20b-4311-a626-ac5d85a58d88 is now active with session: 98afb387-22b9-4baa-bfe2-a8c33ced2aee","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.094011769Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.094030567Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342,"session_id":"98afb387-22b9-4baa-bfe2-a8c33ced2aee"}
{"msg":"The guest has finished waiting for activation with:40","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:01.094043599Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Flush
02/30 Write block 508 len 16384 data: 4 4 4 5
03/30 Write block 109 len 36864 data: 6 5 5 5 3 3 4 6 7
04/30 Write block 32 len 40960 data: 6 6 5 5 4 4 4 3 4 4
05/30 Read block 298 len 24576
06/30 Read block 122 len 16384
07/30 Read block 588 len 28672
08/30 Read block 189 len 4096
09/30 Write block 63 len 32768 data: 4 4 6 5 5 6 5 6
10/30 Read block 408 len 24576
11/30 Write block 389 len 12288 data: 4 4 4
12/30 Write block 564 len 32768 data: 4 5 3 3 3 3 4 4
13/30 Flush
14/30 Write block 409 len 8192 data: 4 3
15/30 Write block 370 len 40960 data: 5 5 5 5 5 5 5 6 7 7
16/30 Read block 277 len 28672
17/30 Read block 265 len 12288
18/30 Read block 403 len 32768
19/30 Write block 196 len 8192 data: 7 8
20/30 Write block 82 len 28672 data: 6 6 7 5 5 4 4
21/30 Write block 274 len 20480 data: 6 7 6 7 7
22/30 Flush
23/30 Write block 142 len 36864 data: 4 4 4 6 7 7 7 7 6
24/30 Read block 200 len 36864
25/30 Read block 346 len 24576
26/30 Write block 40 len 16384 data: 5 5 4 3
27/30 Write block 469 len 16384 data: 4 4 3 3
28/30 Read block 473 len 40960
29/30 Read block 95 len 8192
30/30 Read block 186 len 20480
----------------------------------------------------------------
Crucible gen:40 GIO:true work queues: Upstairs:0 downstairs:8
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1022 Acked Write 9 Done Done Done false
1023 Acked Read 9 Done Done Done false
1024 Acked Read 6 Done Done Done false
1025 Acked Write 4 Done Done Done false
1026 Acked Write 4 Done Done Done false
1027 Acked Read 10 Done Done Done false
1028 Acked Read 2 Done Done Done false
1029 Acked Read 5 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 1 2
Done 8 7 7 22
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1021 1021 1021
Downstairs last five completed: 1021 1020 1019 1018 1017
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:01.114991677Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1342}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 36 36 36 61 61 61 F F F
1 020-039 40 40 40 67 67 67 F F F
2 040-059 40 40 40 67 67 67 T T T
3 060-079 40 40 40 67 67 67 F F F
4 080-099 40 40 40 68 68 68 F F F
5 100-119 40 40 40 67 67 67 F F F
6 120-139 39 39 39 65 65 65 F F F
7 140-159 39 39 39 65 65 65 T T T
8 160-179 39 39 39 65 65 65 F F F
9 180-199 40 40 40 68 68 68 F F F
10 200-219 35 35 35 59 59 59 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 40 40 40 68 68 68 F F F
14 280-299 39 39 39 65 65 65 F F F
15 300-319 39 39 39 65 65 65 F F F
16 320-339 36 36 36 61 61 61 F F F
17 340-359 37 37 37 62 62 62 F F F
18 360-379 40 40 40 68 68 68 F F F
19 380-399 40 40 40 67 67 67 F F F
20 400-419 40 40 40 68 68 68 F F F
21 420-439 33 33 33 56 56 56 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 32 32 32 55 55 55 T T T
24 480-499 39 39 39 65 65 65 F F F
25 500-519 40 40 40 67 67 67 F F F
26 520-539 36 36 36 60 60 60 F F F
27 540-559 36 36 36 60 60 60 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 39 39 39 65 65 65 F F F
Max gen: 40, Max flush: 68
A Difference in extent metadata is expected here
On loop 20
Verifying data now
No change after verify
Loop: 20 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 36 36 36 61 61 61 F F F
1 020-039 40 40 40 67 67 67 F F F
2 040-059 41 41 41 69 69 69 F F F
3 060-079 40 40 40 67 67 67 F F F
4 080-099 40 40 40 68 68 68 F F F
5 100-119 40 40 40 67 67 67 F F F
6 120-139 39 39 39 65 65 65 F F F
7 140-159 41 41 41 69 69 69 F F F
8 160-179 39 39 39 65 65 65 F F F
9 180-199 40 40 40 68 68 68 F F F
10 200-219 35 35 35 59 59 59 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 40 40 40 68 68 68 F F F
14 280-299 39 39 39 65 65 65 F F F
15 300-319 39 39 39 65 65 65 F F F
16 320-339 36 36 36 61 61 61 F F F
17 340-359 37 37 37 62 62 62 F F F
18 360-379 40 40 40 68 68 68 F F F
19 380-399 40 40 40 67 67 67 F F F
20 400-419 40 40 40 68 68 68 F F F
21 420-439 33 33 33 56 56 56 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 41 41 41 69 69 69 F F F
24 480-499 39 39 39 65 65 65 F F F
25 500-519 40 40 40 67 67 67 F F F
26 520-539 36 36 36 60 60 60 F F F
27 540-559 36 36 36 60 60 60 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 39 39 39 65 65 65 F F F
Max gen: 41, Max flush: 69
Begin loop 21
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.362576408Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.409111551Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.410210641Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.410254263Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.410269634Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.410284065Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350}
{"msg":"Crucible ebe55191-9840-4dd0-b46c-85d81b9c2460 has session id: 1f658a0e-e994-4607-ab59-d90b89e1562c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.410944996Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"Upstairs opts: Upstairs UUID: ebe55191-9840-4dd0-b46c-85d81b9c2460, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411120197Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"Crucible stats registered with UUID: ebe55191-9840-4dd0-b46c-85d81b9c2460","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411151578Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:05.411166518Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411655176Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"task":"crutest"}
{"msg":"The guest has requested activation with gen:42","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411678572Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350}
{"msg":"ebe55191-9840-4dd0-b46c-85d81b9c2460 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411746889Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411763729Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"0","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411787714Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"0","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411801345Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"1","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411814997Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"1","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411827749Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"2","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411841321Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"2","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411854293Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"io task","client":"2","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411878278Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"io task","client":"0","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.411904002Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"io task","client":"1","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:43314)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.412058546Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"io task","client":"1","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57958)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.412128463Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"io task","client":"2","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:34653)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.412202817Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"io task","client":"0","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.412710173Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"1","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.412738605Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"0","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.412960478Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"2","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.413465975Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"1","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.413493228Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"0","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.413552481Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"2","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.41405532Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414163463Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"[0]R flush_numbers[0..12]: [61, 67, 69, 67, 68, 67, 65, 69, 65, 68, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414206056Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"downstairs","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"[0]R generation[0..12]: [36, 40, 41, 40, 40, 40, 39, 41, 39, 40, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414224015Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"downstairs","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414238196Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"downstairs","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"[1]R flush_numbers[0..12]: [61, 67, 69, 67, 68, 67, 65, 69, 65, 68, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414251728Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"downstairs","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"[1]R generation[0..12]: [36, 40, 41, 40, 40, 40, 39, 41, 39, 40, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.41426506Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"downstairs","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414282069Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"downstairs","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"[2]R flush_numbers[0..12]: [61, 67, 69, 67, 68, 67, 65, 69, 65, 68, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414295611Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"downstairs","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"[2]R generation[0..12]: [36, 40, 41, 40, 40, 40, 39, 41, 39, 40, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414308773Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"downstairs","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414322115Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"downstairs","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"Max found gen is 42","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414352336Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"downstairs","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"Generation requested: 42 >= found:42","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414366477Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"downstairs","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"Next flush: 70","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:38:05.414379049Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"downstairs","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414395769Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"":"downstairs","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414408621Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414420284Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"0","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414433136Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"1","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414445768Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"client":"2","":"downstairs","session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.41445858Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414470643Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"ebe55191-9840-4dd0-b46c-85d81b9c2460 is now active with session: 1f658a0e-e994-4607-ab59-d90b89e1562c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414483075Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414496476Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350,"session_id":"1f658a0e-e994-4607-ab59-d90b89e1562c"}
{"msg":"The guest has finished waiting for activation with:42","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:05.414508519Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 73 len 20480 data: 6 5 4 3 3
02/30 Write block 150 len 20480 data: 7 6 6 9 7
03/30 Write block 486 len 32768 data: 5 6 6 5 5 5 7 6
04/30 Write block 310 len 20480 data: 5 5 4 4 5
05/30 Read block 129 len 32768
06/30 Flush
07/30 Write block 19 len 28672 data: 4 4 4 5 7 7 6
08/30 Read block 329 len 20480
09/30 Write block 85 len 20480 data: 6 6 5 5 4
10/30 Write block 1 len 16384 data: 2 3 6 6
11/30 Write block 434 len 8192 data: 4 4
12/30 Read block 153 len 28672
13/30 Read block 92 len 40960
14/30 Write block 7 len 24576 data: 8 6 6 6 5 3
15/30 Read block 353 len 8192
16/30 Write block 295 len 20480 data: 3 4 6 4 5
17/30 Flush
18/30 Write block 586 len 16384 data: 4 3 3 4
19/30 Write block 546 len 28672 data: 3 4 3 3 2 4 4
20/30 Flush
21/30 Write block 417 len 16384 data: 3 4 4 3
22/30 Read block 576 len 24576
23/30 Flush
24/30 Write block 300 len 16384 data: 6 6 4 6
25/30 Read block 556 len 4096
26/30 Flush
27/30 Write block 306 len 40960 data: 5 7 7 7 6 6 5 5 6 4
28/30 Write block 430 len 28672 data: 4 3 3 3 5 5 4
29/30 Read block 236 len 24576
30/30 Write block 388 len 40960 data: 4 5 5 5 5 4 4 3 2 2
----------------------------------------------------------------
Crucible gen:42 GIO:true work queues: Upstairs:0 downstairs:4
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1026 Acked Write 10 Done Done Done false
1027 Acked Write 7 Done Done Done false
1028 Acked Read 6 Done Done Done false
1029 Acked Write 10 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 3 3 3 9
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1025 1025 1025
Downstairs last five completed: 1025 1024 1023 1022 1021
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:05.434094181Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1350}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 42 42 42 71 71 71 F F F
2 040-059 41 41 41 69 69 69 F F F
3 060-079 42 42 42 70 70 70 F F F
4 080-099 42 42 42 71 71 71 F F F
5 100-119 40 40 40 67 67 67 F F F
6 120-139 39 39 39 65 65 65 F F F
7 140-159 42 42 42 70 70 70 F F F
8 160-179 39 39 39 65 65 65 F F F
9 180-199 40 40 40 68 68 68 F F F
10 200-219 35 35 35 59 59 59 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 40 40 40 68 68 68 F F F
14 280-299 42 42 42 71 71 71 F F F
15 300-319 42 42 42 74 74 74 T T T
16 320-339 36 36 36 61 61 61 F F F
17 340-359 37 37 37 62 62 62 F F F
18 360-379 40 40 40 68 68 68 F F F
19 380-399 40 40 40 67 67 67 T T T
20 400-419 42 42 42 73 73 73 F F F
21 420-439 42 42 42 73 73 73 T T T
22 440-459 26 26 26 45 45 45 F F F
23 460-479 41 41 41 69 69 69 F F F
24 480-499 42 42 42 70 70 70 F F F
25 500-519 40 40 40 67 67 67 F F F
26 520-539 36 36 36 60 60 60 F F F
27 540-559 42 42 42 72 72 72 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 42 42 42 72 72 72 F F F
Max gen: 42, Max flush: 74
A Difference in extent metadata is expected here
On loop 21
Verifying data now
No change after verify
Loop: 21 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 42 42 42 71 71 71 F F F
2 040-059 41 41 41 69 69 69 F F F
3 060-079 42 42 42 70 70 70 F F F
4 080-099 42 42 42 71 71 71 F F F
5 100-119 40 40 40 67 67 67 F F F
6 120-139 39 39 39 65 65 65 F F F
7 140-159 42 42 42 70 70 70 F F F
8 160-179 39 39 39 65 65 65 F F F
9 180-199 40 40 40 68 68 68 F F F
10 200-219 35 35 35 59 59 59 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 23 23 23 39 39 39 F F F
13 260-279 40 40 40 68 68 68 F F F
14 280-299 42 42 42 71 71 71 F F F
15 300-319 43 43 43 75 75 75 F F F
16 320-339 36 36 36 61 61 61 F F F
17 340-359 37 37 37 62 62 62 F F F
18 360-379 40 40 40 68 68 68 F F F
19 380-399 43 43 43 75 75 75 F F F
20 400-419 42 42 42 73 73 73 F F F
21 420-439 43 43 43 75 75 75 F F F
22 440-459 26 26 26 45 45 45 F F F
23 460-479 41 41 41 69 69 69 F F F
24 480-499 42 42 42 70 70 70 F F F
25 500-519 40 40 40 67 67 67 F F F
26 520-539 36 36 36 60 60 60 F F F
27 540-559 42 42 42 72 72 72 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 42 42 42 72 72 72 F F F
Max gen: 43, Max flush: 75
Begin loop 22
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.681731089Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:09.72647672Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.727132834Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.75882302Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.759591784Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.75963095Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.759649778Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.759664289Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358}
{"msg":"Crucible 5af4f3ec-14b0-441f-9291-9ff439330cba has session id: 7261c6d1-524b-4f83-9e35-8ca966addab2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.760418752Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"Upstairs opts: Upstairs UUID: 5af4f3ec-14b0-441f-9291-9ff439330cba, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.76045485Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"Crucible stats registered with UUID: 5af4f3ec-14b0-441f-9291-9ff439330cba","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.76047022Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:09.760482802Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.760895577Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"task":"crutest"}
{"msg":"The guest has requested activation with gen:44","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.760920152Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358}
{"msg":"5af4f3ec-14b0-441f-9291-9ff439330cba active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.760979355Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.760995965Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"0","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.76102011Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"0","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.761034141Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"1","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.761048352Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"1","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.761096133Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"2","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.761115741Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"2","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.761129392Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"io task","client":"2","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.761143133Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"io task","client":"0","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.761268576Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"io task","client":"1","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53713)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.761308841Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"io task","client":"0","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"ds_connection connected from Ok(127.0.0.1:63910)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.761528785Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"io task","client":"1","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47712)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.761564343Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"io task","client":"2","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.762026756Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"1","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.762057338Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"2","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.762282418Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"0","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.76276307Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"1","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.762791703Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"2","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.762817647Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"0","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.76336163Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763461398Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"[0]R flush_numbers[0..12]: [71, 71, 69, 70, 71, 67, 65, 70, 65, 68, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763556079Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"downstairs","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"[0]R generation[0..12]: [42, 42, 41, 42, 42, 40, 39, 42, 39, 40, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763593286Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"downstairs","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763608277Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"downstairs","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"[1]R flush_numbers[0..12]: [71, 71, 69, 70, 71, 67, 65, 70, 65, 68, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763621649Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"downstairs","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"[1]R generation[0..12]: [42, 42, 41, 42, 42, 40, 39, 42, 39, 40, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.76363487Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"downstairs","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763648402Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"downstairs","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"[2]R flush_numbers[0..12]: [71, 71, 69, 70, 71, 67, 65, 70, 65, 68, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763663033Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"downstairs","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"[2]R generation[0..12]: [42, 42, 41, 42, 42, 40, 39, 42, 39, 40, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763676395Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"downstairs","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763690066Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"downstairs","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"Max found gen is 44","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763702639Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"downstairs","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"Generation requested: 44 >= found:44","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763715281Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"downstairs","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"Next flush: 76","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763727703Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"downstairs","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763740845Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"":"downstairs","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763753587Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763768068Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"0","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.76378131Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,Wait for a query_work_queue command to finish before sending IO
"client":"1","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763802317Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"client":"2","":"downstairs","session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763815239Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763827051Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"5af4f3ec-14b0-441f-9291-9ff439330cba is now active with session: 7261c6d1-524b-4f83-9e35-8ca966addab2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763839554Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763851227Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358,"session_id":"7261c6d1-524b-4f83-9e35-8ca966addab2"}
{"msg":"The guest has finished waiting for activation with:44","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:09.763863009Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 257 len 4096 data: 3
02/30 Write block 187 len 36864 data: 5 3 3 3 5 4 4 5 8
03/30 Flush
04/30 Write block 455 len 36864 data: 3 3 3 3 3 4 4 3 4
05/30 Read block 239 len 28672
06/30 Read block 47 len 36864
07/30 Read block 32 len 32768
08/30 Read block 180 len 20480
09/30 Read block 541 len 32768
10/30 Read block 377 len 32768
11/30 Flush
12/30 Write block 424 len 40960 data: 4 4 3 3 3 5 5 4 4 4
13/30 Read block 248 len 40960
14/30 Read block 228 len 32768
15/30 Write block 102 len 28672 data: 5 5 6 7 6 5 5
16/30 Read block 111 len 40960
17/30 Write block 64 len 32768 data: 5 7 6 6 7 6 7 7
18/30 Read block 174 len 20480
19/30 Write block 596 len 12288 data: 3 3 3
20/30 Write block 516 len 16384 data: 4 4 4 4
21/30 Read block 3 len 24576
22/30 Read block 78 len 36864
23/30 Read block 197 len 28672
24/30 Read block 241 len 16384
25/30 Write block 256 len 40960 data: 3 4 2 2 3 3 3 4 4 4
26/30 Read block 21 len 28672
27/30 Read block 342 len 32768
28/30 Read block 504 len 40960
29/30 Read block 257 len 36864
30/30 Read block 417 len 36864
----------------------------------------------------------------
Crucible gen:44 GIO:true work queues: Upstairs:0 downstairs:19
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1011 Acked Write 10 Done Done Done false
1012 Acked Read 10 Done Done Done false
1013 Acked Read 8 Done Done Done false
1014 Acked Write 7 Done Done Done false
1015 Acked Read 10 Done Done Done false
1016 Acked Write 8 Done Done Done false
1017 Acked Read 5 Done Done Done false
1018 Acked Write 3 Done Done Done false
1019 Acked Write 4 Done Done Done false
1020 Acked Read 6 Done Done Done false
1021 Acked Read 9 Done Done Done false
1022 Acked Read 7 Done Done Done false
1023 Acked Read 4 Done Done Done false
1024 Acked Write 10 Done Done Done false
1025 Acked Read 7 Done Done Done false
1026 Acked Read 8 Done Done Done false
1027 Acked Read 10 Done Done Done false
1028 Acked Read 9 Done Done Done false
1029 Acked Read 9 Sent Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 0 1 2
Done 18 19 18 55
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1010 1010 1010
Downstairs last five completed: 1010 1009 1008 1007 1006
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:09.787344828Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1358}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 42 42 42 71 71 71 F F F
2 040-059 41 41 41 69 69 69 F F F
3 060-079 42 42 42 70 70 70 T T T
4 080-099 42 42 42 71 71 71 F F F
5 100-119 40 40 40 67 67 67 T T T
6 120-139 39 39 39 65 65 65 F F F
7 140-159 42 42 42 70 70 70 F F F
8 160-179 39 39 39 65 65 65 F F F
9 180-199 44 44 44 76 76 76 F F F
10 200-219 35 35 35 59 59 59 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 44 44 44 76 76 76 T T T
13 260-279 40 40 40 68 68 68 T T T
14 280-299 42 42 42 71 71 71 F F F
15 300-319 43 43 43 75 75 75 F F F
16 320-339 36 36 36 61 61 61 F F F
17 340-359 37 37 37 62 62 62 F F F
18 360-379 40 40 40 68 68 68 F F F
19 380-399 43 43 43 75 75 75 F F F
20 400-419 42 42 42 73 73 73 F F F
21 420-439 43 43 43 75 75 75 T T T
22 440-459 44 44 44 77 77 77 F F F
23 460-479 44 44 44 77 77 77 F F F
24 480-499 42 42 42 70 70 70 F F F
25 500-519 40 40 40 67 67 67 T T T
26 520-539 36 36 36 60 60 60 F F F
27 540-559 42 42 42 72 72 72 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 42 42 42 72 72 72 T T T
Max gen: 44, Max flush: 77
A Difference in extent metadata is expected here
On loop 22
Verifying data now
No change after verify
Loop: 22 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 42 42 42 71 71 71 F F F
2 040-059 41 41 41 69 69 69 F F F
3 060-079 45 45 45 78 78 78 F F F
4 080-099 42 42 42 71 71 71 F F F
5 100-119 45 45 45 78 78 78 F F F
6 120-139 39 39 39 65 65 65 F F F
7 140-159 42 42 42 70 70 70 F F F
8 160-179 39 39 39 65 65 65 F F F
9 180-199 44 44 44 76 76 76 F F F
10 200-219 35 35 35 59 59 59 F F F
11 220-239 23 23 23 39 39 39 F F F
12 240-259 45 45 45 78 78 78 F F F
13 260-279 45 45 45 78 78 78 F F F
14 280-299 42 42 42 71 71 71 F F F
15 300-319 43 43 43 75 75 75 F F F
16 320-339 36 36 36 61 61 61 F F F
17 340-359 37 37 37 62 62 62 F F F
18 360-379 40 40 40 68 68 68 F F F
19 380-399 43 43 43 75 75 75 F F F
20 400-419 42 42 42 73 73 73 F F F
21 420-439 45 45 45 78 78 78 F F F
22 440-459 44 44 44 77 77 77 F F F
23 460-479 44 44 44 77 77 77 F F F
24 480-499 42 42 42 70 70 70 F F F
25 500-519 45 45 45 78 78 78 F F F
26 520-539 36 36 36 60 60 60 F F F
27 540-559 42 42 42 72 72 72 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 45 45 45 78 78 78 F F F
Max gen: 45, Max flush: 78
Begin loop 23
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.085056328Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.129794434Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.130588792Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.130629887Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.130644448Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.130658689Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366}
{"msg":"Crucible 6f5d75c9-262e-4a1d-856a-35bf1783ab4c has session id: ba969a8a-0beb-4395-b135-faf91fd0858d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.131219951Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"Upstairs opts: Upstairs UUID: 6f5d75c9-262e-4a1d-856a-35bf1783ab4c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.131355977Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"Crucible stats registered with UUID: 6f5d75c9-262e-4a1d-856a-35bf1783ab4c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.131385089Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:14.131398181Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.131832022Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"task":"crutest"}
{"msg":"The guest has requested activation with gen:46","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.131858376Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366}
{"msg":"6f5d75c9-262e-4a1d-856a-35bf1783ab4c active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.131890037Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.131903258Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"0","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.131926824Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"0","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.131940455Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"1","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.131999449Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"1","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.132038764Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"2","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.132067636Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"2","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.132145348Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"io task","client":"2","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.132176619Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"io task","client":"0","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.132203212Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"io task","client":"1","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57592)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.132233164Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"io task","client":"2","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57741)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.132376355Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"io task","client":"1","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:38250)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.132413352Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"io task","client":"0","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.132879654Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"0","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.132919239Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"2","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.133541614Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"0","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.133646189Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"2","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.134187914Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.13425808Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.134852892Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"1","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.135362446Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"1","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"[0]R flush_numbers[0..12]: [71, 71, 69, 78, 71, 78, 65, 70, 65, 76, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.135869542Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"downstairs","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"[0]R generation[0..12]: [42, 42, 41, 45, 42, 45, 39, 42, 39, 44, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.135891389Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"downstairs","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.13590617Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"downstairs","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"[1]R flush_numbers[0..12]: [71, 71, 69, 78, 71, 78, 65, 70, 65, 76, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.135920251Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"downstairs","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"[1]R generation[0..12]: [42, 42, 41, 45, 42, 45, 39, 42, 39, 44, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.135933613Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"downstairs","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.135947254Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"downstairs","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"[2]R flush_numbers[0..12]: [71, 71, 69, 78, 71, 78, 65, 70, 65, 76, 59, 39]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.135960376Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"downstairs","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"[2]R generation[0..12]: [42, 42, 41, 45, 42, 45, 39, 42, 39, 44, 35, 23]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.135973398Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"downstairs","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.135992036Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"downstairs","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"Max found gen is 46","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.136005128Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"downstairs","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"Generation requested: 46 >= found:46","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.1360184Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO
downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"Next flush: 79","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.136040056Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"downstairs","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.136052589Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"":"downstairs","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.136065641Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.136077333Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"0","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.136090425Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"1","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.136103148Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"client":"2","":"downstairs","session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.1361162Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.136128032Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"6f5d75c9-262e-4a1d-856a-35bf1783ab4c is now active with session: ba969a8a-0beb-4395-b135-faf91fd0858d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.136140444Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.136151997Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366,"session_id":"ba969a8a-0beb-4395-b135-faf91fd0858d"}
{"msg":"The guest has finished waiting for activation with:46","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:14.13616376Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 343 len 36864 data: 5 6 5 5 4 5 6 5 4
02/30 Read block 33 len 24576
03/30 Read block 124 len 40960
04/30 Write block 548 len 36864 data: 4 4 3 5 5 3 4 5 5
05/30 Write block 415 len 12288 data: 2 2 4
06/30 Read block 353 len 32768
07/30 Flush
08/30 Write block 354 len 40960 data: 6 6 7 7 8 7 7 4 2 2
09/30 Write block 374 len 16384 data: 6 6 6 7
10/30 Read block 114 len 40960
11/30 Write block 97 len 40960 data: 4 4 4 5 5 6 6 7 8 7
12/30 Write block 173 len 28672 data: 4 5 7 7 7 7 8
13/30 Read block 121 len 12288
14/30 Read block 448 len 8192
15/30 Flush
16/30 Write block 374 len 24576 data: 7 7 7 8 8 8
17/30 Write block 370 len 40960 data: 6 6 6 6 8 8 8 9 9 9
18/30 Read block 292 len 24576
19/30 Read block 481 len 16384
20/30 Flush
21/30 Write block 473 len 12288 data: 3 4 4
22/30 Read block 48 len 24576
23/30 Read block 176 len 20480
24/30 Read block 58 len 28672
25/30 Flush
26/30 Write block 215 len 24576 data: 5 3 3 3 2 2
27/30 Read block 498 len 20480
28/30 Write block 519 len 28672 data: 5 5 7 7 7 7 6
29/30 Write block 491 len 12288 data: 6 8 7
30/30 Write block 462 len 16384 data: 4 5 4 3
----------------------------------------------------------------
Crucible gen:46 GIO:true work queues: Upstairs:0 downstairs:5
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1025 Acked Write 6 Done Done Done false
1026 Acked Read 5 Done Done Done false
1027 Acked Write 7 Sent Sent Sent false
1028 Acked Write 3 Sent Sent Sent false
1029 Acked Write 4 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 3 3 3 9
Done 2 2 2 6
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1024 1024 1024
Downstairs last five completed: 1024 1023 1022 1021 1020
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:14.157742887Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1366}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 42 42 42 71 71 71 F F F
2 040-059 41 41 41 69 69 69 F F F
3 060-079 45 45 45 78 78 78 F F F
4 080-099 46 46 46 80 80 80 F F F
5 100-119 46 46 46 80 80 80 F F F
6 120-139 39 39 39 65 65 65 F F F
7 140-159 42 42 42 70 70 70 F F F
8 160-179 46 46 46 80 80 80 F F F
9 180-199 44 44 44 76 76 76 F F F
10 200-219 35 35 35 59 59 59 T T T
11 220-239 23 23 23 39 39 39 T T T
12 240-259 45 45 45 78 78 78 F F F
13 260-279 45 45 45 78 78 78 F F F
14 280-299 42 42 42 71 71 71 F F F
15 300-319 43 43 43 75 75 75 F F F
16 320-339 36 36 36 61 61 61 F F F
17 340-359 46 46 46 80 80 80 F F F
18 360-379 46 46 46 81 81 81 F F F
19 380-399 43 43 43 75 75 75 F F F
20 400-419 46 46 46 79 79 79 F F F
21 420-439 45 45 45 78 78 78 F F F
22 440-459 44 44 44 77 77 77 F F F
23 460-479 46 46 46 82 82 82 T T T
24 480-499 42 42 42 70 70 70 T T T
25 500-519 45 45 45 78 78 78 T T T
26 520-539 36 36 36 60 60 60 T T T
27 540-559 46 46 46 79 79 79 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 45 45 45 78 78 78 F F F
Max gen: 46, Max flush: 82
A Difference in extent metadata is expected here
On loop 23
Verifying data now
No change after verify
Loop: 23 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 42 42 42 71 71 71 F F F
2 040-059 41 41 41 69 69 69 F F F
3 060-079 45 45 45 78 78 78 F F F
4 080-099 46 46 46 80 80 80 F F F
5 100-119 46 46 46 80 80 80 F F F
6 120-139 39 39 39 65 65 65 F F F
7 140-159 42 42 42 70 70 70 F F F
8 160-179 46 46 46 80 80 80 F F F
9 180-199 44 44 44 76 76 76 F F F
10 200-219 47 47 47 83 83 83 F F F
11 220-239 47 47 47 83 83 83 F F F
12 240-259 45 45 45 78 78 78 F F F
13 260-279 45 45 45 78 78 78 F F F
14 280-299 42 42 42 71 71 71 F F F
15 300-319 43 43 43 75 75 75 F F F
16 320-339 36 36 36 61 61 61 F F F
17 340-359 46 46 46 80 80 80 F F F
18 360-379 46 46 46 81 81 81 F F F
19 380-399 43 43 43 75 75 75 F F F
20 400-419 46 46 46 79 79 79 F F F
21 420-439 45 45 45 78 78 78 F F F
22 440-459 44 44 44 77 77 77 F F F
23 460-479 47 47 47 83 83 83 F F F
24 480-499 47 47 47 83 83 83 F F F
25 500-519 47 47 47 83 83 83 F F F
26 520-539 47 47 47 83 83 83 F F F
27 540-559 46 46 46 79 79 79 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 45 45 45 78 78 78 F F F
Max gen: 47, Max flush: 83
Begin loop 24
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.42039223Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:18.465069264Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.465730455Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.497692973Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.498652488Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.498690005Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.498705036Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.498719677Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374}
{"msg":"Crucible 92a3d58e-768a-4b6c-938a-dcb877774a47 has session id: 57ad041f-9874-4c12-a1c1-093ac380f5f9","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.499424081Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"Upstairs opts: Upstairs UUID: 92a3d58e-768a-4b6c-938a-dcb877774a47, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.499474479Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"Crucible stats registered with UUID: 92a3d58e-768a-4b6c-938a-dcb877774a47","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.499503781Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:18.499523919Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.499940511Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"task":"crutest"}
{"msg":"The guest has requested activation with gen:48","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.499971242Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374}
{"msg":"92a3d58e-768a-4b6c-938a-dcb877774a47 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.500045925Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.500072469Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"0","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.500098163Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"0","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.500112364Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"1","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.500126885Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"1","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.500148961Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"2","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.500175165Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"2","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.500199021Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"io task","client":"2","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.500225244Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"io task","client":"0","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.500252357Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"io task","client":"1","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"ds_connection connected from Ok(127.0.0.1:40646)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.500322774Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"io task","client":"1","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"ds_connection connected from Ok(127.0.0.1:55897)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.500356323Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"io task","client":"2","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"ds_connection connected from Ok(127.0.0.1:33879)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.500381477Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"io task","client":"0","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.501250519Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"1","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.501279521Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"2","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.501294582Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"0","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.501954654Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"1","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502135911Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"0","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502166582Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"2","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502487484Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502721818Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"[0]R flush_numbers[0..12]: [71, 71, 69, 78, 80, 80, 65, 70, 80, 76, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502843033Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"downstairs","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"[0]R generation[0..12]: [42, 42, 41, 45, 46, 46, 39, 42, 46, 44, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502862341Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"downstairs","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502876872Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"downstairs","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"[1]R flush_numbers[0..12]: [71, 71, 69, 78, 80, 80, 65, 70, 80, 76, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502894951Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"downstairs","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"[1]R generation[0..12]: [42, 42, 41, 45, 46, 46, 39, 42, 46, 44, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502909392Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"downstairs","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502923394Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"downstairs","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"[2]R flush_numbers[0..12]: [71, 71, 69, 78, 80, 80, 65, 70, 80, 76, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502936875Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"downstairs","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"[2]R generation[0..12]: [42, 42, 41, 45, 46, 46, 39, 42, 46, 44, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502950087Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"downstairs","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502963449Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"downstairs","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"Max found gen is 48","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502975901Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"downstairs","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"Generation requested: 48 >= found:48","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.502988513Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"downstairs","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"Next flush: 84","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:38:18.503001035Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"downstairs","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.503018974Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"":"downstairs","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.503031756Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.503043399Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"0","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.503056221Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"1","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.503068784Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"client":"2","":"downstairs","session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.503083115Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.503095337Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"92a3d58e-768a-4b6c-938a-dcb877774a47 is now active with session: 57ad041f-9874-4c12-a1c1-093ac380f5f9","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.503107609Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.503119002Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374,"session_id":"57ad041f-9874-4c12-a1c1-093ac380f5f9"}
{"msg":"The guest has finished waiting for activation with:48","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:18.503130875Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 95 len 12288 data: 5 5 5
02/30 Read block 307 len 24576
03/30 Read block 427 len 40960
04/30 Read block 136 len 4096
05/30 Read block 388 len 4096
06/30 Flush
07/30 Write block 466 len 4096 data: 3
08/30 Read block 25 len 40960
09/30 Read block 584 len 4096
10/30 Write block 193 len 12288 data: 5 6 9
11/30 Read block 91 len 12288
12/30 Write block 101 len 4096 data: 6
13/30 Read block 194 len 12288
14/30 Write block 451 len 4096 data: 5
15/30 Write block 585 len 8192 data: 4 5
16/30 Read block 308 len 16384
17/30 Write block 45 len 4096 data: 3
18/30 Read block 595 len 16384
19/30 Flush
20/30 Write block 260 len 40960 data: 4 4 4 5 5 5 4 3 4 5
21/30 Write block 274 len 36864 data: 7 8 7 8 8 6 5 4 4
22/30 Read block 25 len 12288
23/30 Write block 483 len 28672 data: 3 4 4 6 7 7 6
24/30 Flush
25/30 Write block 141 len 24576 data: 4 5 5 5 7 8
26/30 Read block 252 len 24576
27/30 Write block 39 len 20480 data: 4 6 6 5 4
28/30 Write block 297 len 24576 data: 7 5 6 7 7 5
29/30 Read block 306 len 24576
30/30 Write block 554 len 12288 data: 5 6 6
----------------------------------------------------------------
Crucible gen:48 GIO:true work queues: Upstairs:0 downstairs:6
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1024 Acked Write 6 Done Done Done false
1025 Acked Read 6 Done Done Done false
1026 Acked Write 5 Done Done Done false
1027 Acked Write 6 Done Done Done false
1028 Acked Read 6 Done Done Done false
1029 Acked Write 3 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 5 5 5 15
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1023 1023 1023
Downstairs last five completed: 1023 1022 1021 1020 1019
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:18.522428725Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1374}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 42 42 42 71 71 71 T T T
2 040-059 48 48 48 85 85 85 T T T
3 060-079 45 45 45 78 78 78 F F F
4 080-099 48 48 48 84 84 84 F F F
5 100-119 48 48 48 85 85 85 F F F
6 120-139 39 39 39 65 65 65 F F F
7 140-159 42 42 42 70 70 70 T T T
8 160-179 46 46 46 80 80 80 F F F
9 180-199 48 48 48 85 85 85 F F F
10 200-219 47 47 47 83 83 83 F F F
11 220-239 47 47 47 83 83 83 F F F
12 240-259 45 45 45 78 78 78 F F F
13 260-279 48 48 48 86 86 86 F F F
14 280-299 48 48 48 86 86 86 T T T
15 300-319 43 43 43 75 75 75 T T T
16 320-339 36 36 36 61 61 61 F F F
17 340-359 46 46 46 80 80 80 F F F
18 360-379 46 46 46 81 81 81 F F F
19 380-399 43 43 43 75 75 75 F F F
20 400-419 46 46 46 79 79 79 F F F
21 420-439 45 45 45 78 78 78 F F F
22 440-459 48 48 48 85 85 85 F F F
23 460-479 48 48 48 85 85 85 F F F
24 480-499 48 48 48 86 86 86 F F F
25 500-519 47 47 47 83 83 83 F F F
26 520-539 47 47 47 83 83 83 F F F
27 540-559 46 46 46 79 79 79 T T T
28 560-579 40 40 40 67 67 67 F F F
29 580-599 48 48 48 85 85 85 F F F
Max gen: 48, Max flush: 86
A Difference in extent metadata is expected here
On loop 24
Verifying data now
No change after verify
Loop: 24 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 49 49 49 87 87 87 F F F
2 040-059 49 49 49 87 87 87 F F F
3 060-079 45 45 45 78 78 78 F F F
4 080-099 48 48 48 84 84 84 F F F
5 100-119 48 48 48 85 85 85 F F F
6 120-139 39 39 39 65 65 65 F F F
7 140-159 49 49 49 87 87 87 F F F
8 160-179 46 46 46 80 80 80 F F F
9 180-199 48 48 48 85 85 85 F F F
10 200-219 47 47 47 83 83 83 F F F
11 220-239 47 47 47 83 83 83 F F F
12 240-259 45 45 45 78 78 78 F F F
13 260-279 48 48 48 86 86 86 F F F
14 280-299 49 49 49 87 87 87 F F F
15 300-319 49 49 49 87 87 87 F F F
16 320-339 36 36 36 61 61 61 F F F
17 340-359 46 46 46 80 80 80 F F F
18 360-379 46 46 46 81 81 81 F F F
19 380-399 43 43 43 75 75 75 F F F
20 400-419 46 46 46 79 79 79 F F F
21 420-439 45 45 45 78 78 78 F F F
22 440-459 48 48 48 85 85 85 F F F
23 460-479 48 48 48 85 85 85 F F F
24 480-499 48 48 48 86 86 86 F F F
25 500-519 47 47 47 83 83 83 F F F
26 520-539 47 47 47 83 83 83 F F F
27 540-559 49 49 49 87 87 87 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 48 48 48 85 85 85 F F F
Max gen: 49, Max flush: 87
Begin loop 25
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.817365524Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.863060188Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.864142338Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.864183643Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.86420351Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.86422074Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382}
{"msg":"Crucible c6534474-79b5-4ee8-b992-5e3c179a0fa5 has session id: 40d15c33-d6c6-4284-9a71-254dc6b2b16f","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.864907574Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"Upstairs opts: Upstairs UUID: c6534474-79b5-4ee8-b992-5e3c179a0fa5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.864957813Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"Crucible stats registered with UUID: c6534474-79b5-4ee8-b992-5e3c179a0fa5","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.864988994Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:22.865017566Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.865524712Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"task":"crutest"}
{"msg":"The guest has requested activation with gen:50","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.865554084Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382}
{"msg":"c6534474-79b5-4ee8-b992-5e3c179a0fa5 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.86562446Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.865644768Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"0","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.865674789Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"0","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.865692808Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"1","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.865710337Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"1","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.865726767Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"2","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.865761765Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"2","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.865788858Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"io task","client":"2","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.865813423Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"io task","client":"0","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.865839647Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"io task","client":"1","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"ds_connection connected from Ok(127.0.0.1:42044)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.86586649Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"io task","client":"2","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"ds_connection connected from Ok(127.0.0.1:60458)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.865942483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"io task","client":"1","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"ds_connection connected from Ok(127.0.0.1:33216)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.865989854Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"io task","client":"0","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.866597778Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"1","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.866751622Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"0","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.867145568Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"1","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.867266114Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"0","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.867795866Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.867916891Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.868333913Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"2","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.868860387Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"2","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"[0]R flush_numbers[0..12]: [71, 87, 87, 78, 84, 85, 65, 87, 80, 85, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869372909Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"downstairs","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"[0]R generation[0..12]: [42, 49, 49, 45, 48, 48, 39, 49, 46, 48, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869400622Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"downstairs","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869415923Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"downstairs","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"[1]R flush_numbers[0..12]: [71, 87, 87, 78, 84, 85, 65, 87, 80, 85, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869429284Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"downstairs","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"[1]R generation[0..12]: [42, 49, 49, 45, 48, 48, 39, 49, 46, 48, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869443106Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"downstairs","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869456777Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"downstairs","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"[2]R flush_numbers[0..12]: [71, 87, 87, 78, 84, 85, 65, 87, 80, 85, 83, 83]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869470109Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"downstairs","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"[2]R generation[0..12]: [42, 49, 49, 45, 48, 48, 39, 49, 46, 48, 47, 47]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869483601Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"downstairs","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869497092Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"downstairs","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"Max found gen is 50","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869509744Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","Wait for a query_work_queue command to finish before sending IO
pid":1382,"":"downstairs","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"Generation requested: 50 >= found:50","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869528882Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"downstairs","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"Next flush: 88","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869541575Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"downstairs","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869554157Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"":"downstairs","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869571406Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869583409Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"0","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869596431Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"1","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869608983Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"client":"2","":"downstairs","session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869621815Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869633668Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"c6534474-79b5-4ee8-b992-5e3c179a0fa5 is now active with session: 40d15c33-d6c6-4284-9a71-254dc6b2b16f","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.86964592Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:22.869657663Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382,"session_id":"40d15c33-d6c6-4284-9a71-254dc6b2b16f"}
{"msg":"The guest has finished waiting for activation with:50","v":0,"name":"crucible","level":30,Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
"time":"2025-01-27T19:38:22.869669466Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 288 len 36864 data: 3 3 3 4 5 5 4 4 5
02/30 Read block 257 len 20480
03/30 Read block 305 len 40960
04/30 Read block 181 len 12288
05/30 Read block 258 len 40960
06/30 Write block 74 len 16384 data: 6 5 4 4
07/30 Read block 345 len 4096
08/30 Read block 401 len 8192
09/30 Write block 319 len 12288 data: 4 4 4
10/30 Write block 445 len 24576 data: 4 4 4 4 5 5
11/30 Read block 394 len 8192
12/30 Read block 182 len 20480
13/30 Read block 351 len 24576
14/30 Read block 132 len 20480
15/30 Write block 269 len 40960 data: 6 5 6 6 5 8 9 8 9 9
16/30 Write block 113 len 40960 data: 4 4 5 7 8 6 6 5 5 4
17/30 Read block 564 len 16384
18/30 Read block 420 len 16384
19/30 Write block 373 len 16384 data: 7 9 9 9
20/30 Write block 395 len 16384 data: 4 3 3 2
21/30 Write block 24 len 32768 data: 8 7 7 5 7 7 8 7
22/30 Flush
23/30 Write block 386 len 40960 data: 4 4 5 6 6 6 6 5 5 5
24/30 Write block 175 len 36864 data: 8 8 8 8 9 8 9 8 6
25/30 Read block 274 len 40960
26/30 Read block 426 len 20480
27/30 Write block 432 len 4096 data: 5
28/30 Write block 455 len 4096 data: 4
29/30 Read block 545 len 8192
30/30 Write block 216 len 28672 data: 4 4 4 3 3 2 2
----------------------------------------------------------------
Crucible gen:50 GIO:true work queues: Upstairs:0 downstairs:8
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1022 Acked Write 10 Done Done Done false
1023 Acked Write 9 Done Done Done false
1024 Acked Read 10 Done Done Done false
1025 Acked Read 5 Done Done Done false
1026 Acked Write 1 Done Done Done false
1027 Acked Write 1 Done Done Done false
1028 Acked Read 2 Done Done Done false
1029 Acked Write 7 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 7 7 7 21
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1021 1021 1021
Downstairs last five completed: 1021 1020 1019 1018 1017
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:22.890692427Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1382}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 50 50 50 88 88 88 F F F
2 040-059 49 49 49 87 87 87 F F F
3 060-079 50 50 50 88 88 88 F F F
4 080-099 48 48 48 84 84 84 F F F
5 100-119 50 50 50 88 88 88 F F F
6 120-139 50 50 50 88 88 88 F F F
7 140-159 49 49 49 87 87 87 F F F
8 160-179 46 46 46 80 80 80 T T T
9 180-199 48 48 48 85 85 85 T T T
10 200-219 47 47 47 83 83 83 T T T
11 220-239 47 47 47 83 83 83 T T T
12 240-259 45 45 45 78 78 78 F F F
13 260-279 50 50 50 88 88 88 F F F
14 280-299 50 50 50 88 88 88 F F F
15 300-319 50 50 50 88 88 88 F F F
16 320-339 50 50 50 88 88 88 F F F
17 340-359 46 46 46 80 80 80 F F F
18 360-379 50 50 50 88 88 88 F F F
19 380-399 50 50 50 88 88 88 T T T
20 400-419 46 46 46 79 79 79 F F F
21 420-439 45 45 45 78 78 78 T T T
22 440-459 50 50 50 88 88 88 T T T
23 460-479 48 48 48 85 85 85 F F F
24 480-499 48 48 48 86 86 86 F F F
25 500-519 47 47 47 83 83 83 F F F
26 520-539 47 47 47 83 83 83 F F F
27 540-559 49 49 49 87 87 87 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 48 48 48 85 85 85 F F F
Max gen: 50, Max flush: 88
A Difference in extent metadata is expected here
On loop 25
Verifying data now
No change after verify
Loop: 25 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 50 50 50 88 88 88 F F F
2 040-059 49 49 49 87 87 87 F F F
3 060-079 50 50 50 88 88 88 F F F
4 080-099 48 48 48 84 84 84 F F F
5 100-119 50 50 50 88 88 88 F F F
6 120-139 50 50 50 88 88 88 F F F
7 140-159 49 49 49 87 87 87 F F F
8 160-179 51 51 51 89 89 89 F F F
9 180-199 51 51 51 89 89 89 F F F
10 200-219 51 51 51 89 89 89 F F F
11 220-239 51 51 51 89 89 89 F F F
12 240-259 45 45 45 78 78 78 F F F
13 260-279 50 50 50 88 88 88 F F F
14 280-299 50 50 50 88 88 88 F F F
15 300-319 50 50 50 88 88 88 F F F
16 320-339 50 50 50 88 88 88 F F F
17 340-359 46 46 46 80 80 80 F F F
18 360-379 50 50 50 88 88 88 F F F
19 380-399 51 51 51 89 89 89 F F F
20 400-419 46 46 46 79 79 79 F F F
21 420-439 51 51 51 89 89 89 F F F
22 440-459 51 51 51 89 89 89 F F F
23 460-479 48 48 48 85 85 85 F F F
24 480-499 48 48 48 86 86 86 F F F
25 500-519 47 47 47 83 83 83 F F F
26 520-539 47 47 47 83 83 83 F F F
27 540-559 49 49 49 87 87 87 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 48 48 48 85 85 85 F F F
Max gen: 51, Max flush: 89
Begin loop 26
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.162237061Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.207782138Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.20913506Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.209179972Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.209194283Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.209208125Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390}
{"msg":"Crucible 8d3478e6-2299-4c56-a051-a447e5ae14a2 has session id: 69dadf69-aa90-4487-8a28-d7d6b229a6bd","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.209980167Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"Upstairs opts: Upstairs UUID: 8d3478e6-2299-4c56-a051-a447e5ae14a2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.210031055Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"Crucible stats registered with UUID: 8d3478e6-2299-4c56-a051-a447e5ae14a2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.210057648Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:27.210078746Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.21060446Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"task":"crutest"}
{"msg":"The guest has requested activation with gen:52","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.210631143Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390}
{"msg":"8d3478e6-2299-4c56-a051-a447e5ae14a2 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.21070166Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.210724286Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"0","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.210750679Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"0","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.210764791Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"1","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.210778532Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"1","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.210798889Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"2","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.210827122Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"2","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.210854575Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"io task","client":"2","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.210883077Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"io task","client":"0","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.210908432Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"io task","client":"1","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57342)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.210931378Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"io task","client":"2","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:32880)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.211118651Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"io task","client":"1","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"ds_connection connected from Ok(127.0.0.1:54264)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.211159966Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"io task","client":"0","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.211617593Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"1","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.21175146Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"0","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.211883908Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"2","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.21229983Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"1","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.212327173Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"0","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.212630925Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"2","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.212975901Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213031617Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"[0]R flush_numbers[0..12]: [71, 88, 87, 88, 84, 88, 88, 87, 89, 89, 89, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213213145Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"downstairs","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"[0]R generation[0..12]: [42, 50, 49, 50, 48, 50, 50, 49, 51, 51, 51, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213240328Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"downstairs","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213255618Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"downstairs","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"[1]R flush_numbers[0..12]: [71, 88, 87, 88, 84, 88, 88, 87, 89, 89, 89, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213270149Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"downstairs","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"[1]R generation[0..12]: [42, 50, 49, 50, 48, 50, 50, 49, 51, 51, 51, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213283841Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"downstairs","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213297702Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"downstairs","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"[2]R flush_numbers[0..12]: [71, 88, 87, 88, 84, 88, 88, 87, 89, 89, 89, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213311084Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"downstairs","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"[2]R generation[0..12]: [42, 50, 49, 50, 48, 50, 50, 49, 51, 51, 51, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213324266Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"downstairs","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO
2025-01-27T19:38:27.213337717Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"downstairs","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"Max found gen is 52","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213360413Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"downstairs","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"Generation requested: 52 >= found:52","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213373135Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"downstairs","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"Next flush: 90","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213385638Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"downstairs","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.21339801Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"":"downstairs","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213410862Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213422405Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"0","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213435177Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"1","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213447769Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"client":"2","":"downstairs","session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213460561Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213472394Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"8d3478e6-2299-4c56-a051-a447e5ae14a2 is now active with session: 69dadf69-aa90-4487-8a28-d7d6b229a6bd","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213484756Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213496509Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390,"session_id":"69dadf69-aa90-4487-8a28-d7d6b229a6bd"}
{"msg":"The guest has finished waiting for activation with:52","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:27.213508282Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 100 len 36864 data: 6 7 7 7 8 9 8 6 6
02/30 Read block 345 len 40960
03/30 Write block 82 len 20480 data: 7 7 8 7 7
04/30 Read block 360 len 40960
05/30 Write block 552 len 12288 data: 6 4 6
06/30 Write block 535 len 40960 data: 8 6 5 4 3 3 4 3 3 3
07/30 Read block 510 len 24576
08/30 Read block 105 len 20480
09/30 Read block 327 len 8192
10/30 Read block 580 len 32768
11/30 Write block 358 len 8192 data: 9 8
12/30 Read block 127 len 8192
13/30 Read block 454 len 12288
14/30 Read block 477 len 28672
15/30 Write block 466 len 12288 data: 4 3 3
16/30 Read block 596 len 16384
17/30 Write block 390 len 28672 data: 7 7 7 6 6 6 4
18/30 Read block 482 len 12288
19/30 Write block 477 len 16384 data: 4 5 3 4
20/30 Write block 339 len 16384 data: 6 6 5 5
21/30 Write block 434 len 8192 data: 6 6
22/30 Read block 251 len 32768
23/30 Write block 134 len 28672 data: 5 4 5 5 5 5 5
24/30 Write block 60 len 32768 data: 3 3 4 5 6 8 7 7
25/30 Read block 3 len 20480
26/30 Write block 208 len 8192 data: 7 6
27/30 Write block 408 len 28672 data: 5 5 4 3 3 4 4
28/30 Read block 409 len 12288
29/30 Write block 39 len 28672 data: 5 7 7 6 5 4 4
30/30 Write block 131 len 20480 data: 3 3 4 6 5
----------------------------------------------------------------
Crucible gen:52 GIO:true work queues: Upstairs:0 downstairs:30
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Write 9 Done Done Done false
1001 Acked Read 10 Done Done Done false
1002 Acked Write 5 Done Done Done false
1003 Acked Read 10 Done Done Done false
1004 Acked Write 3 Done Done Done false
1005 Acked Write 10 Done Done Done false
1006 Acked Read 6 Done Done Done false
1007 Acked Read 5 Done Done Done false
1008 Acked Read 2 Done Done Done false
1009 Acked Read 8 Done Done Done false
1010 Acked Write 2 Done Done Done false
1011 Acked Read 2 Done Done Done false
1012 Acked Read 3 Done Done Done false
1013 Acked Read 7 Done Done Done false
1014 Acked Write 3 Done Done Done false
1015 Acked Read 4 Done Done Done false
1016 Acked Write 7 Done Done Done false
1017 Acked Read 3 Done Done Done false
1018 Acked Write 4 Done Done Done false
1019 Acked Write 4 Done Done Done false
1020 Acked Write 2 Done Done Done false
1021 Acked Read 8 Done Done Done false
1022 Acked Write 7 Done Done Done false
1023 Acked Write 8 Done Done Done false
1024 Acked Read 5 Done Done Done false
1025 Acked Write 2 Done Done Done false
1026 Acked Write 7 Done Done Done false
1027 Acked Read 3 Done Done Done false
1028 Acked Write 7 Sent Sent Sent false
1029 Acked Write 5 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 28 28 28 84
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 0 0 0
Downstairs last five completed:
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:27.234433904Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1390}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 50 50 50 88 88 88 T T T
2 040-059 49 49 49 87 87 87 T T T
3 060-079 50 50 50 88 88 88 T T T
4 080-099 48 48 48 84 84 84 T T T
5 100-119 50 50 50 88 88 88 T T T
6 120-139 50 50 50 88 88 88 T T T
7 140-159 49 49 49 87 87 87 T T T
8 160-179 51 51 51 89 89 89 F F F
9 180-199 51 51 51 89 89 89 F F F
10 200-219 51 51 51 89 89 89 T T T
11 220-239 51 51 51 89 89 89 F F F
12 240-259 45 45 45 78 78 78 F F F
13 260-279 50 50 50 88 88 88 F F F
14 280-299 50 50 50 88 88 88 F F F
15 300-319 50 50 50 88 88 88 F F F
16 320-339 50 50 50 88 88 88 T T T
17 340-359 46 46 46 80 80 80 T T T
18 360-379 50 50 50 88 88 88 F F F
19 380-399 51 51 51 89 89 89 T T T
20 400-419 46 46 46 79 79 79 T T T
21 420-439 51 51 51 89 89 89 T T T
22 440-459 51 51 51 89 89 89 F F F
23 460-479 48 48 48 85 85 85 T T T
24 480-499 48 48 48 86 86 86 T T T
25 500-519 47 47 47 83 83 83 F F F
26 520-539 47 47 47 83 83 83 T T T
27 540-559 49 49 49 87 87 87 T T T
28 560-579 40 40 40 67 67 67 F F F
29 580-599 48 48 48 85 85 85 F F F
Max gen: 51, Max flush: 89
A Difference in extent metadata is expected here
On loop 26
Verifying data now
No change after verify
Loop: 26 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 52 52 52 90 90 90 F F F
2 040-059 52 52 52 90 90 90 F F F
3 060-079 52 52 52 90 90 90 F F F
4 080-099 52 52 52 90 90 90 F F F
5 100-119 52 52 52 90 90 90 F F F
6 120-139 52 52 52 90 90 90 F F F
7 140-159 52 52 52 90 90 90 F F F
8 160-179 51 51 51 89 89 89 F F F
9 180-199 51 51 51 89 89 89 F F F
10 200-219 52 52 52 90 90 90 F F F
11 220-239 51 51 51 89 89 89 F F F
12 240-259 45 45 45 78 78 78 F F F
13 260-279 50 50 50 88 88 88 F F F
14 280-299 50 50 50 88 88 88 F F F
15 300-319 50 50 50 88 88 88 F F F
16 320-339 52 52 52 90 90 90 F F F
17 340-359 52 52 52 90 90 90 F F F
18 360-379 50 50 50 88 88 88 F F F
19 380-399 52 52 52 90 90 90 F F F
20 400-419 52 52 52 90 90 90 F F F
21 420-439 52 52 52 90 90 90 F F F
22 440-459 51 51 51 89 89 89 F F F
23 460-479 52 52 52 90 90 90 F F F
24 480-499 52 52 52 90 90 90 F F F
25 500-519 47 47 47 83 83 83 F F F
26 520-539 52 52 52 90 90 90 F F F
27 540-559 52 52 52 90 90 90 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 48 48 48 85 85 85 F F F
Max gen: 52, Max flush: 90
Begin loop 27
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.558692648Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:41.602612481Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.603302144Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.63544441Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.636554452Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.636596486Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.636616324Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.636632234Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398}
{"msg":"Crucible d3b9d9e9-8d97-483d-aa93-7d8bf0e964af has session id: c4411888-66ef-49e9-847e-31af695def90","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.637374135Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"Upstairs opts: Upstairs UUID: d3b9d9e9-8d97-483d-aa93-7d8bf0e964af, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.63741411Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"Crucible stats registered with UUID: d3b9d9e9-8d97-483d-aa93-7d8bf0e964af","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.637431079Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:41.637444941Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.63789833Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"task":"crutest"}
{"msg":"The guest has requested activation with gen:54","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.637935437Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398}
{"msg":"d3b9d9e9-8d97-483d-aa93-7d8bf0e964af active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.63799534Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.638023123Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"0","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.638054413Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"0","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.638079878Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"1","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.638109449Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"1","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.638146097Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"2","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.63817379Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"2","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.638200693Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"io task","client":"2","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.638228925Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"io task","client":"0","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.638257078Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"io task","client":"1","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52929)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.638287189Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"io task","client":"0","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"ds_connection connected from Ok(127.0.0.1:58673)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.63830229Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"io task","client":"1","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"ds_connection connected from Ok(127.0.0.1:46678)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.638336179Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"io task","client":"2","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.639087873Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"1","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.63912608Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"2","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.639382271Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"0","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.63991728Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"1","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.639946192Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"2","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640009053Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"0","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.64051499Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640621334Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"[0]R flush_numbers[0..12]: [71, 90, 90, 90, 90, 90, 90, 90, 89, 89, 90, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640715046Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"downstairs","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"[0]R generation[0..12]: [42, 52, 52, 52, 52, 52, 52, 52, 51, 51, 52, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640734044Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"downstairs","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640748695Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"downstairs","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"[1]R flush_numbers[0..12]: [71, 90, 90, 90, 90, 90, 90, 90, 89, 89, 90, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640763046Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"downstairs","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"[1]R generation[0..12]: [42, 52, 52, 52, 52, 52, 52, 52, 51, 51, 52, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640777467Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"downstairs","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640792598Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"downstairs","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"[2]R flush_numbers[0..12]: [71, 90, 90, 90, 90, 90, 90, 90, 89, 89, 90, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640806569Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"downstairs","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"[2]R generation[0..12]: [42, 52, 52, 52, 52, 52, 52, 52, 51, 51, 52, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.6408205Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"downstairs","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640834872Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"downstairs","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"Max found gen is 53","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640847974Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"downstairs","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"Generation requested: 54 >= found:53","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:38:41.640861255Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"downstairs","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"Next flush: 91","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640878564Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"downstairs","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640891647Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"":"downstairs","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640909535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640922118Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"0","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640935719Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"1","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640949011Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"client":"2","":"downstairs","session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640962373Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640974665Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"d3b9d9e9-8d97-483d-aa93-7d8bf0e964af is now active with session: c4411888-66ef-49e9-847e-31af695def90","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640987667Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.640999749Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398,"session_id":"c4411888-66ef-49e9-847e-31af695def90"}
{"msg":"The guest has finished waiting for activation with:54","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:41.641012102Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 335 len 4096 data: 3
02/30 Write block 286 len 40960 data: 2 2 4 4 4 5 6 6 5 5
03/30 Write block 549 len 36864 data: 5 4 6 7 5 7 7 7 6
04/30 Flush
05/30 Write block 393 len 4096 data: 7
06/30 Write block 153 len 28672 data: 10 8 4 5 3 3 3
07/30 Read block 132 len 40960
08/30 Flush
09/30 Write block 411 len 8192 data: 4 4
10/30 Write block 116 len 40960 data: 8 9 7 7 6 6 5 2 2 2
11/30 Read block 321 len 40960
12/30 Read block 194 len 12288
13/30 Flush
14/30 Write block 323 len 40960 data: 7 8 9 7 8 6 5 4 3 4
15/30 Write block 392 len 8192 data: 8 8
16/30 Write block 306 len 40960 data: 6 8 8 8 7 7 6 6 7 5
17/30 Read block 330 len 36864
18/30 Read block 125 len 12288
19/30 Write block 419 len 8192 data: 5 4
20/30 Flush
21/30 Write block 550 len 8192 data: 5 7
22/30 Write block 161 len 40960 data: 4 3 3 5 5 5 4 4 3 4
23/30 Read block 587 len 20480
24/30 Read block 33 len 28672
25/30 Write block 50 len 24576 data: 4 5 6 6 6 6
26/30 Read block 579 len 16384
27/30 Read block 370 len 12288
28/30 Read block 535 len 40960
29/30 Write block 353 len 40960 data: 6 7 7 8 8 10 9 8 5 3
30/30 Write block 403 len 12288 data: 5 5 5
----------------------------------------------------------------
Crucible gen:54 GIO:true work queues: Upstairs:0 downstairs:10
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1020 Acked Write 2 Done Done Done false
1021 Acked Write 10 Done Done Done false
1022 Acked Read 5 Done Done Done false
1023 Acked Read 7 Done Done Done false
1024 Acked Write 6 Done Done Done false
1025 Acked Read 4 Done Done Done false
1026 Acked Read 3 Done Done Done false
1027 Acked Read 10 Done Done Done false
1028 Acked Write 10 Sent Sent Sent false
1029 Acked Write 3 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 8 8 8 24
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1019 1019 1019
Downstairs last five completed: 1019 1018 1017 1016 1015
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:41.66107253Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1398}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 52 52 52 90 90 90 F F F
2 040-059 52 52 52 90 90 90 T T T
3 060-079 52 52 52 90 90 90 F F F
4 080-099 52 52 52 90 90 90 F F F
5 100-119 54 54 54 93 93 93 F F F
6 120-139 54 54 54 93 93 93 F F F
7 140-159 54 54 54 92 92 92 F F F
8 160-179 51 51 51 89 89 89 T T T
9 180-199 51 51 51 89 89 89 F F F
10 200-219 52 52 52 90 90 90 F F F
11 220-239 51 51 51 89 89 89 F F F
12 240-259 45 45 45 78 78 78 F F F
13 260-279 50 50 50 88 88 88 F F F
14 280-299 54 54 54 91 91 91 F F F
15 300-319 54 54 54 94 94 94 F F F
16 320-339 54 54 54 94 94 94 F F F
17 340-359 52 52 52 90 90 90 T T T
18 360-379 50 50 50 88 88 88 T T T
19 380-399 54 54 54 94 94 94 F F F
20 400-419 54 54 54 94 94 94 T T T
21 420-439 54 54 54 94 94 94 F F F
22 440-459 51 51 51 89 89 89 F F F
23 460-479 52 52 52 90 90 90 F F F
24 480-499 52 52 52 90 90 90 F F F
25 500-519 47 47 47 83 83 83 F F F
26 520-539 52 52 52 90 90 90 F F F
27 540-559 54 54 54 91 91 91 T T T
28 560-579 40 40 40 67 67 67 F F F
29 580-599 48 48 48 85 85 85 F F F
Max gen: 54, Max flush: 94
A Difference in extent metadata is expected here
On loop 27
Verifying data now
No change after verify
Loop: 27 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 52 52 52 90 90 90 F F F
2 040-059 55 55 55 95 95 95 F F F
3 060-079 52 52 52 90 90 90 F F F
4 080-099 52 52 52 90 90 90 F F F
5 100-119 54 54 54 93 93 93 F F F
6 120-139 54 54 54 93 93 93 F F F
7 140-159 54 54 54 92 92 92 F F F
8 160-179 55 55 55 95 95 95 F F F
9 180-199 51 51 51 89 89 89 F F F
10 200-219 52 52 52 90 90 90 F F F
11 220-239 51 51 51 89 89 89 F F F
12 240-259 45 45 45 78 78 78 F F F
13 260-279 50 50 50 88 88 88 F F F
14 280-299 54 54 54 91 91 91 F F F
15 300-319 54 54 54 94 94 94 F F F
16 320-339 54 54 54 94 94 94 F F F
17 340-359 55 55 55 95 95 95 F F F
18 360-379 55 55 55 95 95 95 F F F
19 380-399 54 54 54 94 94 94 F F F
20 400-419 55 55 55 95 95 95 F F F
21 420-439 54 54 54 94 94 94 F F F
22 440-459 51 51 51 89 89 89 F F F
23 460-479 52 52 52 90 90 90 F F F
24 480-499 52 52 52 90 90 90 F F F
25 500-519 47 47 47 83 83 83 F F F
26 520-539 52 52 52 90 90 90 F F F
27 540-559 55 55 55 95 95 95 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 48 48 48 85 85 85 F F F
Max gen: 55, Max flush: 95
Begin loop 28
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:45.964779866Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:46.011054251Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.011740746Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.043863165Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.04509773Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.045149008Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.045167087Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.045185406Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406}
{"msg":"Crucible ae2d14b2-3b71-4204-abb6-5611a36310c2 has session id: 118d7d5a-cc3d-4876-889c-8e398dd1d962","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.045838271Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"Upstairs opts: Upstairs UUID: ae2d14b2-3b71-4204-abb6-5611a36310c2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.04590427Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"Crucible stats registered with UUID: ae2d14b2-3b71-4204-abb6-5611a36310c2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.045935111Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:46.045955978Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.046475557Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"task":"crutest"}
{"msg":"The guest has requested activation with gen:56","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.046507167Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406}
{"msg":"ae2d14b2-3b71-4204-abb6-5611a36310c2 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.046595593Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.04661483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"0","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.046640425Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"0","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.046655485Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"1","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.046670956Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"1","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.046685677Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"2","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.046710262Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"2","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.046737435Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"io task","client":"2","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.046753984Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"io task","client":"0","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.046780448Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"io task","client":"1","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39132)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.046813308Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"io task","client":"0","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47827)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.046924509Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"io task","client":"1","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53994)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.047253265Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"io task","client":"2","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.047641065Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"1","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.047819684Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"2","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.04805233Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"0","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.048222065Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"1","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.048325671Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"2","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.048718158Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"0","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.048900555Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.048934514Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"[0]R flush_numbers[0..12]: [71, 90, 95, 90, 90, 93, 93, 92, 95, 89, 90, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049261192Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"downstairs","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"[0]R generation[0..12]: [42, 52, 55, 52, 52, 54, 54, 54, 55, 51, 52, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049282099Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"downstairs","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.04929653Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"downstairs","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"[1]R flush_numbers[0..12]: [71, 90, 95, 90, 90, 93, 93, 92, 95, 89, 90, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049310471Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"downstairs","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"[1]R generation[0..12]: [42, 52, 55, 52, 52, 54, 54, 54, 55, 51, 52, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049324113Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"downstairs","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049337784Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"downstairs","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"[2]R flush_numbers[0..12]: [71, 90, 95, 90, 90, 93, 93, 92, 95, 89, 90, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049354784Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"downstairs","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"[2]R generation[0..12]: [42, 52, 55, 52, 52, 54, 54, 54, 55, 51, 52, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049368256Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"downstairs","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049381707Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"downstairs","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"Max found gen is 56","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049394279Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"downstairs","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"Generation requested: 56 >= found:56","v":0,"name":"crucible","level":30,Wait for a query_work_queue command to finish before sending IO
"time":"2025-01-27T19:38:46.049406941Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"downstairs","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"Next flush: 96","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.04942583Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"downstairs","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049438362Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"":"downstairs","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049451364Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049463107Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"0","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049475869Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"1","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049489151Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"client":"2","":"downstairs","session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049501953Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049514125Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"ae2d14b2-3b71-4204-abb6-5611a36310c2 is now active with session: 118d7d5a-cc3d-4876-889c-8e398dd1d962","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049526458Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049537931Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406,"session_id":"118d7d5a-cc3d-4876-889c-8e398dd1d962"}
{"msg":"The guest has finished waiting for activation with:56","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:46.049549703Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 463 len 12288 data: 6 5 4
02/30 Flush
03/30 Write block 242 len 28672 data: 3 4 4 4 3 3 3
04/30 Write block 26 len 8192 data: 8 6
05/30 Read block 182 len 24576
06/30 Read block 225 len 28672
07/30 Read block 103 len 40960
08/30 Read block 207 len 4096
09/30 Write block 462 len 28672 data: 5 7 6 5 5 4 4
10/30 Read block 20 len 32768
11/30 Write block 296 len 24576 data: 6 8 6 7 8 8
12/30 Read block 243 len 28672
13/30 Write block 93 len 36864 data: 7 6 6 6 6 5 5 7 8
14/30 Read block 251 len 36864
15/30 Read block 390 len 16384
16/30 Read block 444 len 36864
17/30 Read block 538 len 8192
18/30 Flush
19/30 Write block 315 len 12288 data: 6 4 4
20/30 Write block 362 len 40960 data: 4 3 2 2 2 3 3 5 7 7
21/30 Read block 64 len 20480
22/30 Write block 535 len 28672 data: 9 7 6 5 4 4 5
23/30 Write block 435 len 8192 data: 7 5
24/30 Read block 302 len 40960
25/30 Write block 495 len 4096 data: 8
26/30 Write block 269 len 12288 data: 7 6 7
27/30 Write block 166 len 32768 data: 6 5 5 4 5 6 6 5
28/30 Read block 524 len 8192
29/30 Write block 80 len 28672 data: 5 6 8 8 9 8 8
30/30 Write block 303 len 40960 data: 7 5 5 7 9 9 9 8 8 7
----------------------------------------------------------------
Crucible gen:56 GIO:true work queues: Upstairs:0 downstairs:12
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1018 Acked Write 3 Done Done Done false
1019 Acked Write 10 Done Done Done false
1020 Acked Read 5 Done Done Done false
1021 Acked Write 7 Done Done Done false
1022 Acked Write 2 Done Done Done false
1023 Acked Read 10 Done Done Done false
1024 Acked Write 1 Done Done Done false
1025 Acked Write 3 Done Done Done false
1026 Acked Write 8 Done Done Done false
1027 Acked Read 2 Done Done Done false
1028 Acked Write 7 Done Done Done false
1029 Acked Write 10 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 11 11 11 33
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1017 1017 1017
Downstairs last five completed: 1017 1016 1015 1014 1013
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:46.0704344Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1406}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 56 56 56 97 97 97 F F F
2 040-059 55 55 55 95 95 95 F F F
3 060-079 52 52 52 90 90 90 F F F
4 080-099 56 56 56 97 97 97 T T T
5 100-119 56 56 56 97 97 97 F F F
6 120-139 54 54 54 93 93 93 F F F
7 140-159 54 54 54 92 92 92 F F F
8 160-179 55 55 55 95 95 95 T T T
9 180-199 51 51 51 89 89 89 F F F
10 200-219 52 52 52 90 90 90 F F F
11 220-239 51 51 51 89 89 89 F F F
12 240-259 56 56 56 97 97 97 F F F
13 260-279 50 50 50 88 88 88 T T T
14 280-299 56 56 56 97 97 97 F F F
15 300-319 56 56 56 97 97 97 T T T
16 320-339 54 54 54 94 94 94 F F F
17 340-359 55 55 55 95 95 95 F F F
18 360-379 55 55 55 95 95 95 T T T
19 380-399 54 54 54 94 94 94 F F F
20 400-419 55 55 55 95 95 95 F F F
21 420-439 54 54 54 94 94 94 T T T
22 440-459 51 51 51 89 89 89 F F F
23 460-479 56 56 56 97 97 97 F F F
24 480-499 52 52 52 90 90 90 T T T
25 500-519 47 47 47 83 83 83 F F F
26 520-539 52 52 52 90 90 90 T T T
27 540-559 55 55 55 95 95 95 T T T
28 560-579 40 40 40 67 67 67 F F F
29 580-599 48 48 48 85 85 85 F F F
Max gen: 56, Max flush: 97
A Difference in extent metadata is expected here
On loop 28
Verifying data now
No change after verify
Loop: 28 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 56 56 56 97 97 97 F F F
2 040-059 55 55 55 95 95 95 F F F
3 060-079 52 52 52 90 90 90 F F F
4 080-099 57 57 57 98 98 98 F F F
5 100-119 56 56 56 97 97 97 F F F
6 120-139 54 54 54 93 93 93 F F F
7 140-159 54 54 54 92 92 92 F F F
8 160-179 57 57 57 98 98 98 F F F
9 180-199 51 51 51 89 89 89 F F F
10 200-219 52 52 52 90 90 90 F F F
11 220-239 51 51 51 89 89 89 F F F
12 240-259 56 56 56 97 97 97 F F F
13 260-279 57 57 57 98 98 98 F F F
14 280-299 56 56 56 97 97 97 F F F
15 300-319 57 57 57 98 98 98 F F F
16 320-339 54 54 54 94 94 94 F F F
17 340-359 55 55 55 95 95 95 F F F
18 360-379 57 57 57 98 98 98 F F F
19 380-399 54 54 54 94 94 94 F F F
20 400-419 55 55 55 95 95 95 F F F
21 420-439 57 57 57 98 98 98 F F F
22 440-459 51 51 51 89 89 89 F F F
23 460-479 56 56 56 97 97 97 F F F
24 480-499 57 57 57 98 98 98 F F F
25 500-519 47 47 47 83 83 83 F F F
26 520-539 57 57 57 98 98 98 F F F
27 540-559 57 57 57 98 98 98 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 48 48 48 85 85 85 F F F
Max gen: 57, Max flush: 98
Begin loop 29
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.387632884Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.433977205Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.435024487Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.435092275Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.435110304Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.435125994Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414}
{"msg":"Crucible f5492856-04cf-4d09-851f-01ec6f0a64c5 has session id: e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.435827349Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"Upstairs opts: Upstairs UUID: f5492856-04cf-4d09-851f-01ec6f0a64c5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.435873871Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"Crucible stats registered with UUID: f5492856-04cf-4d09-851f-01ec6f0a64c5","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.435900334Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:50.435921171Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436321514Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"task":"crutest"}
{"msg":"The guest has requested activation with gen:58","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436347957Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414}
{"msg":"f5492856-04cf-4d09-851f-01ec6f0a64c5 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436396387Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436424939Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"0","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436495216Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"0","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436530124Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"1","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436612493Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"1","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436644234Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"2","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436670997Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"2","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436694852Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"io task","client":"2","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436764589Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"io task","client":"0","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436898026Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"io task","client":"1","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:41470)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436928878Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"io task","client":"2","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:41300)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436950394Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"io task","client":"0","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:48264)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.436997945Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"io task","client":"1","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.43745747Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"0","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.437543107Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"2","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.437575267Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"1","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.438118201Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"1","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.438267059Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"0","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.438318307Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"2","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.438784828Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.438917906Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"[0]R flush_numbers[0..12]: [71, 97, 95, 90, 98, 97, 93, 92, 98, 89, 90, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439009109Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"downstairs","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"[0]R generation[0..12]: [42, 56, 55, 52, 57, 56, 54, 54, 57, 51, 52, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439034423Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"downstairs","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439049494Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"downstairs","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"[1]R flush_numbers[0..12]: [71, 97, 95, 90, 98, 97, 93, 92, 98, 89, 90, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439063905Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"downstairs","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"[1]R generation[0..12]: [42, 56, 55, 52, 57, 56, 54, 54, 57, 51, 52, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439077937Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"downstairs","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439091618Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"downstairs","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"[2]R flush_numbers[0..12]: [71, 97, 95, 90, 98, 97, 93, 92, 98, 89, 90, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.43910483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"downstairs","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"[2]R generation[0..12]: [42, 56, 55, 52, 57, 56, 54, 54, 57, 51, 52, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439118272Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"downstairs","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439131713Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"downstairs","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"Max found gen is 58","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439144156Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"downstairs","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"Generation requested: 58 >= found:58","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439156768Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO
downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"Next flush: 99","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439175596Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"downstairs","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439188168Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"":"downstairs","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.43920102Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439212753Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"0","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439225395Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"1","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439241016Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"client":"2","":"downstairs","session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439254407Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.43926623Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"f5492856-04cf-4d09-851f-01ec6f0a64c5 is now active with session: e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439278722Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439290325Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414,"session_id":"e46cc6d6-7ef5-480a-a70c-9d25f4cb0f2d"}
{"msg":"The guest has finished waiting for activation with:58","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:50.439302238Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 477 len 28672 data: 5 6 4 5 4 3 4
02/30 Write block 273 len 36864 data: 6 9 10 9 10 10 7 6 5
03/30 Write block 175 len 40960 data: 9 9 9 9 10 9 10 9 7 5
04/30 Write block 164 len 20480 data: 6 6 7 6 6
05/30 Read block 451 len 8192
06/30 Read block 207 len 12288
07/30 Write block 260 len 28672 data: 5 5 5 6 6 6 5
08/30 Read block 83 len 8192
09/30 Write block 161 len 8192 data: 5 4
10/30 Write block 497 len 28672 data: 7 7 6 4 4 4 4
11/30 Write block 54 len 20480 data: 7 7 5 5 5
12/30 Write block 66 len 4096 data: 8
13/30 Write block 92 len 8192 data: 8 8
14/30 Read block 280 len 20480
15/30 Flush
16/30 Write block 118 len 8192 data: 8 8
17/30 Read block 329 len 40960
18/30 Write block 521 len 32768 data: 8 8 8 8 7 6 9 10
19/30 Read block 489 len 20480
20/30 Read block 438 len 40960
21/30 Flush
22/30 Write block 482 len 16384 data: 4 5 5 5
23/30 Read block 12 len 40960
24/30 Read block 96 len 40960
25/30 Read block 377 len 16384
26/30 Read block 511 len 8192
27/30 Read block 400 len 12288
28/30 Read block 543 len 16384
29/30 Read block 37 len 20480
30/30 Write block 453 len 20480 data: 4 3 5 4 4
----------------------------------------------------------------
Crucible gen:58 GIO:true work queues: Upstairs:0 downstairs:9
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1021 Acked Write 4 Done Done Done false
1022 Acked Read 10 Done Done Done false
1023 Acked Read 10 Done Done Done false
1024 Acked Read 4 Done Done Done false
1025 Acked Read 2 Done Done Done false
1026 Acked Read 3 Done Done Done false
1027 Acked Read 4 Done Done Done false
1028 Acked Read 5 Done Done Done false
1029 Acked Write 5 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 8 8 8 24
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1020 1020 1020
Downstairs last five completed: 1020 1019 1018 1017 1016
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:50.459745458Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1414}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 56 56 56 97 97 97 F F F
2 040-059 58 58 58 99 99 99 F F F
3 060-079 58 58 58 99 99 99 F F F
4 080-099 58 58 58 99 99 99 F F F
5 100-119 58 58 58 100 100 100 F F F
6 120-139 54 54 54 93 93 93 F F F
7 140-159 54 54 54 92 92 92 F F F
8 160-179 58 58 58 99 99 99 F F F
9 180-199 58 58 58 99 99 99 F F F
10 200-219 52 52 52 90 90 90 F F F
11 220-239 51 51 51 89 89 89 F F F
12 240-259 56 56 56 97 97 97 F F F
13 260-279 58 58 58 99 99 99 F F F
14 280-299 58 58 58 99 99 99 F F F
15 300-319 57 57 57 98 98 98 F F F
16 320-339 54 54 54 94 94 94 F F F
17 340-359 55 55 55 95 95 95 F F F
18 360-379 57 57 57 98 98 98 F F F
19 380-399 54 54 54 94 94 94 F F F
20 400-419 55 55 55 95 95 95 F F F
21 420-439 57 57 57 98 98 98 F F F
22 440-459 51 51 51 89 89 89 T T T
23 460-479 58 58 58 99 99 99 F F F
24 480-499 58 58 58 99 99 99 T T T
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 57 57 57 98 98 98 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 48 48 48 85 85 85 F F F
Max gen: 58, Max flush: 100
A Difference in extent metadata is expected here
On loop 29
Verifying data now
No change after verify
Loop: 29 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 56 56 56 97 97 97 F F F
2 040-059 58 58 58 99 99 99 F F F
3 060-079 58 58 58 99 99 99 F F F
4 080-099 58 58 58 99 99 99 F F F
5 100-119 58 58 58 100 100 100 F F F
6 120-139 54 54 54 93 93 93 F F F
7 140-159 54 54 54 92 92 92 F F F
8 160-179 58 58 58 99 99 99 F F F
9 180-199 58 58 58 99 99 99 F F F
10 200-219 52 52 52 90 90 90 F F F
11 220-239 51 51 51 89 89 89 F F F
12 240-259 56 56 56 97 97 97 F F F
13 260-279 58 58 58 99 99 99 F F F
14 280-299 58 58 58 99 99 99 F F F
15 300-319 57 57 57 98 98 98 F F F
16 320-339 54 54 54 94 94 94 F F F
17 340-359 55 55 55 95 95 95 F F F
18 360-379 57 57 57 98 98 98 F F F
19 380-399 54 54 54 94 94 94 F F F
20 400-419 55 55 55 95 95 95 F F F
21 420-439 57 57 57 98 98 98 F F F
22 440-459 59 59 59 101 101 101 F F F
23 460-479 58 58 58 99 99 99 F F F
24 480-499 59 59 59 101 101 101 F F F
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 57 57 57 98 98 98 F F F
28 560-579 40 40 40 67 67 67 F F F
29 580-599 48 48 48 85 85 85 F F F
Max gen: 59, Max flush: 101
Begin loop 30
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.703932834Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.750658249Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.751826126Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.75188344Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.751904897Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.751923725Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422}
{"msg":"Crucible 14ec2932-b94b-43c9-a645-a4a6b662bd47 has session id: b52567b5-5b02-4b1b-9ea8-94ca02d14f01","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.752595799Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"Upstairs opts: Upstairs UUID: 14ec2932-b94b-43c9-a645-a4a6b662bd47, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.752640712Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"Crucible stats registered with UUID: 14ec2932-b94b-43c9-a645-a4a6b662bd47","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.752663487Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:54.752682815Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753154134Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"task":"crutest"}
{"msg":"The guest has requested activation with gen:60","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753183076Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422}
{"msg":"14ec2932-b94b-43c9-a645-a4a6b662bd47 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753252582Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753269542Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"0","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753295286Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"0","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753309178Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"1","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753327516Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"1","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753340658Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"2","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753354579Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"2","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753367291Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"io task","client":"2","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753380803Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"io task","client":"0","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753394015Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"io task","client":"1","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39015)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753568957Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"io task","client":"2","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"ds_connection connected from Ok(127.0.0.1:32898)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753600317Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"io task","client":"0","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"ds_connection connected from Ok(127.0.0.1:38928)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.753779646Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"io task","client":"1","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.754310957Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"2","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.754349174Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"0","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.754571266Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"1","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.755076523Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"2","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.755104336Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"0","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.755124163Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"1","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.755639515Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.755746498Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"[0]R flush_numbers[0..12]: [71, 97, 99, 99, 99, 100, 93, 92, 99, 99, 90, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.755846936Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"downstairs","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"[0]R generation[0..12]: [42, 56, 58, 58, 58, 58, 54, 54, 58, 58, 52, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.755874519Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"downstairs","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.755890209Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"downstairs","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"[1]R flush_numbers[0..12]: [71, 97, 99, 99, 99, 100, 93, 92, 99, 99, 90, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.755903891Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"downstairs","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"[1]R generation[0..12]: [42, 56, 58, 58, 58, 58, 54, 54, 58, 58, 52, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.755925667Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"downstairs","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.755951251Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"downstairs","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"[2]R flush_numbers[0..12]: [71, 97, 99, 99, 99, 100, 93, 92, 99, 99, 90, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.755976866Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"Wait for a query_work_queue command to finish before sending IO
downstairs","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"[2]R generation[0..12]: [42, 56, 58, 58, 58, 58, 54, 54, 58, 58, 52, 51]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.756008206Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"downstairs","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.756028474Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"downstairs","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"Max found gen is 60","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.756041446Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"downstairs","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"Generation requested: 60 >= found:60","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.756054387Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"downstairs","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"Next flush: 102","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.756071417Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"downstairs","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.756084159Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"":"downstairs","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.756096961Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.756108664Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"0","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.756121746Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"1","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.756134518Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"client":"2","":"downstairs","session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.7561472Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.756159642Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"14ec2932-b94b-43c9-a645-a4a6b662bd47 is now active with session: b52567b5-5b02-4b1b-9ea8-94ca02d14f01","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.756172425Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.756184037Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422,"session_id":"b52567b5-5b02-4b1b-9ea8-94ca02d14f01"}
{"msg":"The guest has finished waiting for activation with:60","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:54.75619596Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 104 len 4096 data: 9
02/30 Flush
03/30 Write block 368 len 16384 data: 4 6 8 8
04/30 Read block 339 len 4096
05/30 Read block 463 len 4096
06/30 Write block 314 len 12288 data: 8 7 5
07/30 Read block 523 len 4096
08/30 Write block 161 len 40960 data: 6 5 4 7 7 8 7 7 5 6
09/30 Write block 549 len 4096 data: 6
10/30 Flush
11/30 Write block 85 len 40960 data: 9 9 6 6 5 7 6 9 9 7
12/30 Write block 375 len 40960 data: 10 10 10 10 10 6 4 4 4 4
13/30 Read block 272 len 36864
14/30 Flush
15/30 Write block 580 len 40960 data: 3 3 3 3 3 5 6 4 4 5
16/30 Read block 440 len 20480
17/30 Write block 61 len 32768 data: 4 5 6 7 9 9 8 8
18/30 Write block 218 len 24576 data: 5 4 4 3 3 2
19/30 Flush
20/30 Write block 570 len 36864 data: 5 5 6 6 6 5 5 4 3
21/30 Flush
22/30 Write block 248 len 8192 data: 4 3
23/30 Read block 86 len 36864
24/30 Read block 196 len 36864
25/30 Write block 186 len 32768 data: 4 6 4 4 4 6 5 6
26/30 Read block 434 len 16384
27/30 Read block 241 len 28672
28/30 Read block 57 len 8192
29/30 Read block 263 len 24576
30/30 Read block 452 len 36864
----------------------------------------------------------------
Crucible gen:60 GIO:true work queues: Upstairs:0 downstairs:9
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1021 Acked Write 2 Done Done Done false
1022 Acked Read 9 Done Done Done false
1023 Acked Read 9 Done Done Done false
1024 Acked Write 8 Done Done Done false
1025 Acked Read 4 Done Done Done false
1026 Acked Read 7 Done Done Done false
1027 Acked Read 2 Done Done Done false
1028 Acked Read 6 Done Done Done false
1029 Acked Read 9 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 9 9 9 27
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1020 1020 1020
Downstairs last five completed: 1020 1019 1018 1017 1016
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:54.777519505Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1422}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 56 56 56 97 97 97 F F F
2 040-059 58 58 58 99 99 99 F F F
3 060-079 60 60 60 105 105 105 F F F
4 080-099 60 60 60 104 104 104 F F F
5 100-119 60 60 60 102 102 102 F F F
6 120-139 54 54 54 93 93 93 F F F
7 140-159 54 54 54 92 92 92 F F F
8 160-179 60 60 60 103 103 103 F F F
9 180-199 58 58 58 99 99 99 T T T
10 200-219 60 60 60 105 105 105 F F F
11 220-239 60 60 60 105 105 105 F F F
12 240-259 56 56 56 97 97 97 T T T
13 260-279 58 58 58 99 99 99 F F F
14 280-299 58 58 58 99 99 99 F F F
15 300-319 60 60 60 103 103 103 F F F
16 320-339 54 54 54 94 94 94 F F F
17 340-359 55 55 55 95 95 95 F F F
18 360-379 60 60 60 104 104 104 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 55 55 55 95 95 95 F F F
21 420-439 57 57 57 98 98 98 F F F
22 440-459 59 59 59 101 101 101 F F F
23 460-479 58 58 58 99 99 99 F F F
24 480-499 59 59 59 101 101 101 F F F
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 60 60 60 103 103 103 F F F
28 560-579 60 60 60 106 106 106 F F F
29 580-599 60 60 60 105 105 105 F F F
Max gen: 60, Max flush: 106
A Difference in extent metadata is expected here
On loop 30
Verifying data now
No change after verify
Loop: 30 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 56 56 56 97 97 97 F F F
2 040-059 58 58 58 99 99 99 F F F
3 060-079 60 60 60 105 105 105 F F F
4 080-099 60 60 60 104 104 104 F F F
5 100-119 60 60 60 102 102 102 F F F
6 120-139 54 54 54 93 93 93 F F F
7 140-159 54 54 54 92 92 92 F F F
8 160-179 60 60 60 103 103 103 F F F
9 180-199 61 61 61 107 107 107 F F F
10 200-219 60 60 60 105 105 105 F F F
11 220-239 60 60 60 105 105 105 F F F
12 240-259 61 61 61 107 107 107 F F F
13 260-279 58 58 58 99 99 99 F F F
14 280-299 58 58 58 99 99 99 F F F
15 300-319 60 60 60 103 103 103 F F F
16 320-339 54 54 54 94 94 94 F F F
17 340-359 55 55 55 95 95 95 F F F
18 360-379 60 60 60 104 104 104 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 55 55 55 95 95 95 F F F
21 420-439 57 57 57 98 98 98 F F F
22 440-459 59 59 59 101 101 101 F F F
23 460-479 58 58 58 99 99 99 F F F
24 480-499 59 59 59 101 101 101 F F F
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 60 60 60 103 103 103 F F F
28 560-579 60 60 60 106 106 106 F F F
29 580-599 60 60 60 105 105 105 F F F
Max gen: 61, Max flush: 107
Begin loop 31
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.016687439Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:59.060672881Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.061344865Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.093930097Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.094802916Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.094839594Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.094853645Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.094868056Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430}
{"msg":"Crucible 0275f707-ef79-4f07-8a10-d67b35e668de has session id: b2965ac1-ee89-4cd7-a7df-a18d4ca172c9","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.09558874Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"Upstairs opts: Upstairs UUID: 0275f707-ef79-4f07-8a10-d67b35e668de, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.095634241Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"Crucible stats registered with UUID: 0275f707-ef79-4f07-8a10-d67b35e668de","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.095658517Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:59.095674727Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.09613822Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"task":"crutest"}
{"msg":"The guest has requested activation with gen:62","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.096165133Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430}
{"msg":"0275f707-ef79-4f07-8a10-d67b35e668de active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.09625031Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.096267389Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"0","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.096292184Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"0","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.096305836Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"1","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.096319987Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"1","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.096333079Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"2","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.096346611Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"2","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.096359163Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"io task","client":"2","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.096372424Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"io task","client":"0","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.096385367Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"io task","client":"1","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"ds_connection connected from Ok(127.0.0.1:45843)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.096597445Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"io task","client":"2","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"ds_connection connected from Ok(127.0.0.1:40974)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.096810084Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"io task","client":"0","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"ds_connection connected from Ok(127.0.0.1:60937)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.096849689Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"io task","client":"1","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.097387436Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"1","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.097425112Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"2","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.097578728Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"0","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.098183143Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"1","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.098224018Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"2","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.098253739Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"0","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.098819799Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.098924954Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"[0]R flush_numbers[0..12]: [71, 97, 99, 105, 104, 102, 93, 92, 103, 107, 105, 105]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.098998049Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"downstairs","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"[0]R generation[0..12]: [42, 56, 58, 60, 60, 60, 54, 54, 60, 61, 60, 60]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099017107Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"downstairs","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099031358Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"downstairs","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"[1]R flush_numbers[0..12]: [71, 97, 99, 105, 104, 102, 93, 92, 103, 107, 105, 105]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.09904494Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"downstairs","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"[1]R generation[0..12]: [42, 56, 58, 60, 60, 60, 54, 54, 60, 61, 60, 60]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099058591Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"downstairs","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.09907657Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"downstairs","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"[2]R flush_numbers[0..12]: [71, 97, 99, 105, 104, 102, 93, 92, 103, 107, 105, 105]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099090401Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"downstairs","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"[2]R generation[0..12]: [42, 56, 58, 60, 60, 60, 54, 54, 60, 61, 60, 60]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099103693Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"downstairs","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"Wait for a query_work_queue command to finish before sending IO
crucible","level":30,"time":"2025-01-27T19:38:59.09912501Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"downstairs","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"Max found gen is 62","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099137812Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"downstairs","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"Generation requested: 62 >= found:62","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099150714Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"downstairs","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"Next flush: 108","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099163296Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"downstairs","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099175679Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"":"downstairs","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099188361Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099200013Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"0","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099212936Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"1","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099225578Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"client":"2","":"downstairs","session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.09923825Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099250113Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"0275f707-ef79-4f07-8a10-d67b35e668de is now active with session: b2965ac1-ee89-4cd7-a7df-a18d4ca172c9","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099262745Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099276236Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430,"session_id":"b2965ac1-ee89-4cd7-a7df-a18d4ca172c9"}
{"msg":"The guest has finished waiting for activation with:62","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:38:59.099288199Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 341 len 28672 data: 6 6 6 7 6 6 5
02/30 Write block 32 len 16384 data: 7 7 6 6
03/30 Read block 556 len 24576
04/30 Write block 451 len 20480 data: 6 5 5 4 6
05/30 Read block 87 len 28672
06/30 Flush
07/30 Write block 223 len 8192 data: 3 2
08/30 Write block 579 len 20480 data: 3 4 4 4 4
09/30 Read block 554 len 8192
10/30 Read block 134 len 12288
11/30 Write block 154 len 24576 data: 9 5 6 4 4 4
12/30 Write block 118 len 40960 data: 9 9 7 7 6 3 3 3 2 2
13/30 Read block 477 len 12288
14/30 Read block 196 len 36864
15/30 Read block 355 len 32768
16/30 Flush
17/30 Write block 230 len 12288 data: 3 3 2
18/30 Read block 441 len 12288
19/30 Read block 190 len 16384
20/30 Read block 314 len 32768
21/30 Read block 306 len 32768
22/30 Read block 288 len 32768
23/30 Write block 440 len 40960 data: 3 3 3 4 4 5 5 5 5 6
24/30 Read block 33 len 28672
25/30 Read block 56 len 32768
26/30 Read block 366 len 20480
27/30 Write block 430 len 24576 data: 6 5 6 5 7 8
28/30 Read block 522 len 16384
29/30 Write block 38 len 24576 data: 5 6 8 8 7 6
30/30 Write block 453 len 16384 data: 6 5 7 5
----------------------------------------------------------------
Crucible gen:62 GIO:true work queues: Upstairs:0 downstairs:14
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1016 Acked Write 3 Done Done Done false
1017 Acked Read 3 Done Done Done false
1018 Acked Read 4 Done Done Done false
1019 Acked Read 8 Done Done Done false
1020 Acked Read 8 Done Done Done false
1021 Acked Read 8 Done Done Done false
1022 Acked Write 10 Done Done Done false
1023 Acked Read 7 Done Done Done false
1024 Acked Read 8 Done Done Done false
1025 Acked Read 5 Done Done Done false
1026 Acked Write 6 Done Done Done false
1027 Acked Read 4 Done Done Done false
1028 Acked Write 6 Sent Sent Sent false
1029 Acked Write 4 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 12 12 12 36
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1015 1015 1015
Downstairs last five completed: 1015 1014 1013 1012 1011
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:38:59.121104929Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1430}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 62 62 62 108 108 108 T T T
2 040-059 58 58 58 99 99 99 T T T
3 060-079 60 60 60 105 105 105 F F F
4 080-099 60 60 60 104 104 104 F F F
5 100-119 62 62 62 109 109 109 F F F
6 120-139 62 62 62 109 109 109 F F F
7 140-159 62 62 62 109 109 109 F F F
8 160-179 60 60 60 103 103 103 F F F
9 180-199 61 61 61 107 107 107 F F F
10 200-219 60 60 60 105 105 105 F F F
11 220-239 62 62 62 109 109 109 T T T
12 240-259 61 61 61 107 107 107 F F F
13 260-279 58 58 58 99 99 99 F F F
14 280-299 58 58 58 99 99 99 F F F
15 300-319 60 60 60 103 103 103 F F F
16 320-339 54 54 54 94 94 94 F F F
17 340-359 62 62 62 108 108 108 F F F
18 360-379 60 60 60 104 104 104 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 55 55 55 95 95 95 F F F
21 420-439 57 57 57 98 98 98 T T T
22 440-459 62 62 62 108 108 108 T T T
23 460-479 58 58 58 99 99 99 F F F
24 480-499 59 59 59 101 101 101 F F F
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 60 60 60 103 103 103 F F F
28 560-579 62 62 62 109 109 109 F F F
29 580-599 62 62 62 109 109 109 F F F
Max gen: 62, Max flush: 109
A Difference in extent metadata is expected here
On loop 31
Verifying data now
No change after verify
Loop: 31 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 42 42 42 71 71 71 F F F
1 020-039 63 63 63 110 110 110 F F F
2 040-059 63 63 63 110 110 110 F F F
3 060-079 60 60 60 105 105 105 F F F
4 080-099 60 60 60 104 104 104 F F F
5 100-119 62 62 62 109 109 109 F F F
6 120-139 62 62 62 109 109 109 F F F
7 140-159 62 62 62 109 109 109 F F F
8 160-179 60 60 60 103 103 103 F F F
9 180-199 61 61 61 107 107 107 F F F
10 200-219 60 60 60 105 105 105 F F F
11 220-239 63 63 63 110 110 110 F F F
12 240-259 61 61 61 107 107 107 F F F
13 260-279 58 58 58 99 99 99 F F F
14 280-299 58 58 58 99 99 99 F F F
15 300-319 60 60 60 103 103 103 F F F
16 320-339 54 54 54 94 94 94 F F F
17 340-359 62 62 62 108 108 108 F F F
18 360-379 60 60 60 104 104 104 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 55 55 55 95 95 95 F F F
21 420-439 63 63 63 110 110 110 F F F
22 440-459 63 63 63 110 110 110 F F F
23 460-479 58 58 58 99 99 99 F F F
24 480-499 59 59 59 101 101 101 F F F
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 60 60 60 103 103 103 F F F
28 560-579 62 62 62 109 109 109 F F F
29 580-599 62 62 62 109 109 109 F F F
Max gen: 63, Max flush: 110
Begin loop 32
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.404408911Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.450863615Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.451651417Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.451716227Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.451741991Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.451761569Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438}
{"msg":"Crucible 82dfa9a5-0b13-44b3-9be4-3a225a1052c0 has session id: 6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.452505958Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"Upstairs opts: Upstairs UUID: 82dfa9a5-0b13-44b3-9be4-3a225a1052c0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.452547952Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"Crucible stats registered with UUID: 82dfa9a5-0b13-44b3-9be4-3a225a1052c0","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.452564122Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:03.452577064Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.452972909Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"task":"crutest"}
{"msg":"The guest has requested activation with gen:64","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.452997203Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438}
{"msg":"82dfa9a5-0b13-44b3-9be4-3a225a1052c0 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.453076125Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.453092914Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"0","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.453117069Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"0","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.453130481Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"1","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.453144332Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"1","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.453157064Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"2","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.453170636Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"2","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.453187396Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"io task","client":"2","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.453206904Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"io task","client":"0","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.453231578Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"io task","client":"1","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"ds_connection connected from Ok(127.0.0.1:40750)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.453329428Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"io task","client":"2","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47649)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.4533895Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"io task","client":"1","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"ds_connection connected from Ok(127.0.0.1:37118)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.453525486Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"io task","client":"0","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.454235876Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"1","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.454264669Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"0","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.454280199Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"2","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.455066282Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"2","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.455107907Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"1","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.45523108Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"0","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.455628294Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.455743633Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"[0]R flush_numbers[0..12]: [71, 110, 110, 105, 104, 109, 109, 109, 103, 107, 105, 110]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.455866897Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"downstairs","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"[0]R generation[0..12]: [42, 63, 63, 60, 60, 62, 62, 62, 60, 61, 60, 63]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.455886624Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"downstairs","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.455901765Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"downstairs","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"[1]R flush_numbers[0..12]: [71, 110, 110, 105, 104, 109, 109, 109, 103, 107, 105, 110]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.455916306Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"downstairs","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"[1]R generation[0..12]: [42, 63, 63, 60, 60, 62, 62, 62, 60, 61, 60, 63]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.455930687Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"downstairs","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.455945148Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"downstairs","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"[2]R flush_numbers[0..12]: [71, 110, 110, 105, 104, 109, 109, 109, 103, 107, 105, 110]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.455959119Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"downstairs","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"[2]R generation[0..12]: [42, 63, 63, 60, 60, 62, 62, 62, 60, 61, 60, 63]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.455973161Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"downstairs","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.455987672Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"downstairs","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"Max found gen is 64","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.456001153Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"downstairs","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"Generation requested: 64 >= found:64","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.456014955Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"downstairs","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"Next flush: 111","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.456028007Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pidWait for a query_work_queue command to finish before sending IO
":1438,"":"downstairs","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.456050353Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"":"downstairs","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.456063824Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.456078266Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"0","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.456091987Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"1","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.456105349Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"client":"2","":"downstairs","session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.45611888Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.456131513Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"82dfa9a5-0b13-44b3-9be4-3a225a1052c0 is now active with session: 6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.456144585Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.456156787Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438,"session_id":"6cb6e0a0-7a9f-49a5-95f6-adf25cd5240f"}
{"msg":"The guest has finished waiting for activation with:64","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:03.456169229Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 295 len 8192 data: 6 7
02/30 Read block 328 len 28672
03/30 Read block 519 len 36864
04/30 Write block 95 len 4096 data: 7
05/30 Read block 289 len 8192
06/30 Flush
07/30 Write block 85 len 4096 data: 10
08/30 Flush
09/30 Write block 198 len 28672 data: 8 6 6 5 6 6 7
10/30 Flush
11/30 Flush
12/30 Write block 21 len 28672 data: 5 6 8 9 8 9 7
13/30 Write block 17 len 36864 data: 3 4 5 5 6 7 9 10 9
14/30 Write block 433 len 24576 data: 6 8 9 6 3 3
15/30 Read block 280 len 12288
16/30 Read block 544 len 28672
17/30 Read block 204 len 40960
18/30 Read block 363 len 8192
19/30 Write block 454 len 28672 data: 6 8 6 5 4 4 5
20/30 Flush
21/30 Write block 34 len 8192 data: 7 7
22/30 Read block 373 len 40960
23/30 Read block 272 len 40960
24/30 Write block 55 len 36864 data: 8 6 6 6 4 4 5 6 7
25/30 Write block 546 len 8192 data: 4 5
26/30 Read block 186 len 16384
27/30 Write block 237 len 4096 data: 2
28/30 Read block 437 len 28672
29/30 Read block 196 len 32768
30/30 Read block 37 len 24576
----------------------------------------------------------------
Crucible gen:64 GIO:true work queues: Upstairs:0 downstairs:10
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1020 Acked Write 2 Done Done Done false
1021 Acked Read 10 Done Done Done false
1022 Acked Read 10 Done Done Done false
1023 Acked Write 9 Done Done Done false
1024 Acked Write 2 Done Done Done false
1025 Acked Read 4 Done Done Done false
1026 Acked Write 1 Done Done Done false
1027 Acked Read 7 Done Done Done false
1028 Acked Read 8 Done Done Done false
1029 Acked Read 6 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 1 2
Done 10 9 9 28
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1019 1019 1019
Downstairs last five completed: 1019 1018 1017 1016 1015
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:03.476772411Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1438}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 64 64 64 115 115 115 F F F
1 020-039 64 64 64 115 115 115 T T T
2 040-059 63 63 63 110 110 110 T T T
3 060-079 60 60 60 105 105 105 T T T
4 080-099 64 64 64 112 112 112 F F F
5 100-119 62 62 62 109 109 109 F F F
6 120-139 62 62 62 109 109 109 F F F
7 140-159 62 62 62 109 109 109 F F F
8 160-179 60 60 60 103 103 103 F F F
9 180-199 64 64 64 113 113 113 F F F
10 200-219 64 64 64 113 113 113 F F F
11 220-239 63 63 63 110 110 110 T T T
12 240-259 61 61 61 107 107 107 F F F
13 260-279 58 58 58 99 99 99 F F F
14 280-299 64 64 64 111 111 111 F F F
15 300-319 60 60 60 103 103 103 F F F
16 320-339 54 54 54 94 94 94 F F F
17 340-359 62 62 62 108 108 108 F F F
18 360-379 60 60 60 104 104 104 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 55 55 55 95 95 95 F F F
21 420-439 64 64 64 115 115 115 F F F
22 440-459 64 64 64 115 115 115 F F F
23 460-479 64 64 64 115 115 115 F F F
24 480-499 59 59 59 101 101 101 F F F
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 60 60 60 103 103 103 T T T
28 560-579 62 62 62 109 109 109 F F F
29 580-599 62 62 62 109 109 109 F F F
Max gen: 64, Max flush: 115
A Difference in extent metadata is expected here
On loop 32
Verifying data now
No change after verify
Loop: 32 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 64 64 64 115 115 115 F F F
1 020-039 65 65 65 116 116 116 F F F
2 040-059 65 65 65 116 116 116 F F F
3 060-079 65 65 65 116 116 116 F F F
4 080-099 64 64 64 112 112 112 F F F
5 100-119 62 62 62 109 109 109 F F F
6 120-139 62 62 62 109 109 109 F F F
7 140-159 62 62 62 109 109 109 F F F
8 160-179 60 60 60 103 103 103 F F F
9 180-199 64 64 64 113 113 113 F F F
10 200-219 64 64 64 113 113 113 F F F
11 220-239 65 65 65 116 116 116 F F F
12 240-259 61 61 61 107 107 107 F F F
13 260-279 58 58 58 99 99 99 F F F
14 280-299 64 64 64 111 111 111 F F F
15 300-319 60 60 60 103 103 103 F F F
16 320-339 54 54 54 94 94 94 F F F
17 340-359 62 62 62 108 108 108 F F F
18 360-379 60 60 60 104 104 104 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 55 55 55 95 95 95 F F F
21 420-439 64 64 64 115 115 115 F F F
22 440-459 64 64 64 115 115 115 F F F
23 460-479 64 64 64 115 115 115 F F F
24 480-499 59 59 59 101 101 101 F F F
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 65 65 65 116 116 116 F F F
28 560-579 62 62 62 109 109 109 F F F
29 580-599 62 62 62 109 109 109 F F F
Max gen: 65, Max flush: 116
Begin loop 33
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.730857287Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.776957659Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.778014654Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.778054899Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.77806938Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.77808596Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446}
{"msg":"Crucible fd9b3880-6a46-4cdb-8f6e-3ee379ea4867 has session id: ad0f8e4c-a92c-4f6a-a565-05179eb4c913","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.778791104Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"Upstairs opts: Upstairs UUID: fd9b3880-6a46-4cdb-8f6e-3ee379ea4867, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.778942879Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"Crucible stats registered with UUID: fd9b3880-6a46-4cdb-8f6e-3ee379ea4867","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.778989161Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:07.779023729Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.779610616Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"task":"crutest"}
{"msg":"The guest has requested activation with gen:66","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.779637639Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446}
{"msg":"fd9b3880-6a46-4cdb-8f6e-3ee379ea4867 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.779706277Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.779724326Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"0","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.779754267Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"0","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.779768858Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"1","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.779782899Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"1","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.779795661Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"2","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.779809463Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"2","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.779822065Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"io task","client":"2","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.779835327Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"io task","client":"0","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.779857643Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"io task","client":"1","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"ds_connection connected from Ok(127.0.0.1:34126)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.780097764Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"io task","client":"0","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"ds_connection connected from Ok(127.0.0.1:36747)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.780204918Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"io task","client":"2","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"ds_connection connected from Ok(127.0.0.1:59405)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.780254687Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"io task","client":"1","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.780844122Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"1","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.780886836Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"0","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.78097626Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"2","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.781433327Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"1","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.781476501Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"0","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.781725576Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"2","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782023811Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782134313Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"[0]R flush_numbers[0..12]: [115, 116, 116, 116, 112, 109, 109, 109, 103, 113, 113, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782246324Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"downstairs","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"[0]R generation[0..12]: [64, 65, 65, 65, 64, 62, 62, 62, 60, 64, 64, 65]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782265542Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"downstairs","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782279673Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"downstairs","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"[1]R flush_numbers[0..12]: [115, 116, 116, 116, 112, 109, 109, 109, 103, 113, 113, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782293075Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"downstairs","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"[1]R generation[0..12]: [64, 65, 65, 65, 64, 62, 62, 62, 60, 64, 64, 65]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782307986Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"downstairs","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.78233246Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"downstairs","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"[2]R flush_numbers[0..12]: [115, 116, 116, 116, 112, 109, 109, 109, 103, 113, 113, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782359104Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"downstairs","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"[2]R generation[0..12]: [64, 65, 65, 65, 64, 62, 62, 62, 60, 64, 64, 65]","Wait for a query_work_queue command to finish before sending IO
v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782390535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"downstairs","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782417718Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"downstairs","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"Max found gen is 66","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782445131Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"downstairs","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"Generation requested: 66 >= found:66","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782471334Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"downstairs","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"Next flush: 117","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.78249422Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"downstairs","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782517706Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"":"downstairs","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.78254302Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782563837Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"0","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782578108Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"1","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782615805Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"client":"2","":"downstairs","session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
level":30,"time":"2025-01-27T19:39:07.782642638Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782663286Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"fd9b3880-6a46-4cdb-8f6e-3ee379ea4867 is now active with session: ad0f8e4c-a92c-4f6a-a565-05179eb4c913","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782683663Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.782701712Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446,"session_id":"ad0f8e4c-a92c-4f6a-a565-05179eb4c913"}
{"msg":"The guest has finished waiting for activation with:66","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:07.78272121Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 301 len 20480 data: 9 6 8 6 6
02/30 Write block 111 len 28672 data: 6 6 5 5 6 9 10
03/30 Flush
04/30 Write block 30 len 4096 data: 9
05/30 Flush
06/30 Write block 255 len 12288 data: 4 4 5
07/30 Read block 427 len 20480
08/30 Read block 248 len 12288
09/30 Flush
10/30 Write block 312 len 36864 data: 8 7 9 8 6 5 4 5 5
11/30 Read block 117 len 40960
12/30 Read block 402 len 20480
13/30 Read block 129 len 12288
14/30 Flush
15/30 Write block 254 len 20480 data: 4 5 5 6 3
16/30 Write block 244 len 32768 data: 5 5 4 4 5 4 2 3
17/30 Read block 22 len 8192
18/30 Write block 124 len 32768 data: 4 4 3 3 2 2 2 4
19/30 Write block 115 len 12288 data: 7 10 11
20/30 Read block 97 len 20480
21/30 Read block 113 len 8192
22/30 Flush
23/30 Write block 420 len 36864 data: 5 3 3 5 5 5 4 4 4
24/30 Flush
25/30 Flush
26/30 Write block 285 len 4096 data: 2
27/30 Write block 409 len 16384 data: 6 5 5 5
28/30 Write block 265 len 40960 data: 7 6 4 5 8 7 8 7 7 10
29/30 Write block 204 len 40960 data: 8 8 7 7 8 7 5 5 5 5
30/30 Read block 228 len 28672
----------------------------------------------------------------
Crucible gen:66 GIO:true work queues: Upstairs:0 downstairs:5
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1025 Acked Write 1 Done Done Done false
1026 Acked Write 4 Done Done Done false
1027 Acked Write 10 Done Done Done false
1028 Acked Write 10 Done Sent Sent false
1029 Acked Read 7 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 2 2 4
Done 5 3 3 11
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1024 1024 1024
Downstairs last five completed: 1024 1023 1022 1021 1020
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:07.8026479Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1446}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 64 64 64 115 115 115 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 65 65 65 116 116 116 F F F
3 060-079 65 65 65 116 116 116 F F F
4 080-099 64 64 64 112 112 112 F F F
5 100-119 66 66 66 121 121 121 F F F
6 120-139 66 66 66 121 121 121 F F F
7 140-159 62 62 62 109 109 109 F F F
8 160-179 60 60 60 103 103 103 F F F
9 180-199 64 64 64 113 113 113 F F F
10 200-219 64 64 64 113 113 113 T T T
11 220-239 65 65 65 116 116 116 F F F
12 240-259 66 66 66 121 121 121 F F F
13 260-279 58 58 58 99 99 99 T T T
14 280-299 64 64 64 111 111 111 T T T
15 300-319 66 66 66 120 120 120 F F F
16 320-339 66 66 66 120 120 120 F F F
17 340-359 62 62 62 108 108 108 F F F
18 360-379 60 60 60 104 104 104 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 55 55 55 95 95 95 T T T
21 420-439 66 66 66 122 122 122 F F F
22 440-459 64 64 64 115 115 115 F F F
23 460-479 64 64 64 115 115 115 F F F
24 480-499 59 59 59 101 101 101 F F F
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 65 65 65 116 116 116 F F F
28 560-579 62 62 62 109 109 109 F F F
29 580-599 62 62 62 109 109 109 F F F
Max gen: 66, Max flush: 122
A Difference in extent metadata is expected here
On loop 33
Verifying data now
No change after verify
Loop: 33 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 64 64 64 115 115 115 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 65 65 65 116 116 116 F F F
3 060-079 65 65 65 116 116 116 F F F
4 080-099 64 64 64 112 112 112 F F F
5 100-119 66 66 66 121 121 121 F F F
6 120-139 66 66 66 121 121 121 F F F
7 140-159 62 62 62 109 109 109 F F F
8 160-179 60 60 60 103 103 103 F F F
9 180-199 64 64 64 113 113 113 F F F
10 200-219 67 67 67 123 123 123 F F F
11 220-239 65 65 65 116 116 116 F F F
12 240-259 66 66 66 121 121 121 F F F
13 260-279 67 67 67 123 123 123 F F F
14 280-299 67 67 67 123 123 123 F F F
15 300-319 66 66 66 120 120 120 F F F
16 320-339 66 66 66 120 120 120 F F F
17 340-359 62 62 62 108 108 108 F F F
18 360-379 60 60 60 104 104 104 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 67 67 67 123 123 123 F F F
21 420-439 66 66 66 122 122 122 F F F
22 440-459 64 64 64 115 115 115 F F F
23 460-479 64 64 64 115 115 115 F F F
24 480-499 59 59 59 101 101 101 F F F
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 65 65 65 116 116 116 F F F
28 560-579 62 62 62 109 109 109 F F F
29 580-599 62 62 62 109 109 109 F F F
Max gen: 67, Max flush: 123
Begin loop 34
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.05969829Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.105662017Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.1064169Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.106469877Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.106488485Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.106505675Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454}
{"msg":"Crucible d2a24cf7-6912-4aa0-88ea-a5c352ace24e has session id: 449ccb99-f8af-4f66-b193-b18f1f57d6e8","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107064099Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"Upstairs opts: Upstairs UUID: d2a24cf7-6912-4aa0-88ea-a5c352ace24e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107107422Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"Crucible stats registered with UUID: d2a24cf7-6912-4aa0-88ea-a5c352ace24e","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107133246Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:12.10715996Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107558662Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"task":"crutest"}
{"msg":"The guest has requested activation with gen:68","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107583217Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454}
{"msg":"d2a24cf7-6912-4aa0-88ea-a5c352ace24e active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107656202Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107672872Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"0","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107697976Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"0","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107711998Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"1","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107726329Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"1","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107739381Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"2","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107753092Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"2","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107766054Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"io task","client":"2","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107784583Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"io task","client":"0","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107807878Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"io task","client":"1","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"ds_connection connected from Ok(127.0.0.1:46253)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.107995202Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"io task","client":"2","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"ds_connection connected from Ok(127.0.0.1:42344)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.108110581Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"io task","client":"0","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"ds_connection connected from Ok(127.0.0.1:46882)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.108142771Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"io task","client":"1","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.108684365Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"0","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.108724521Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"1","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.109199736Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"2","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.109278947Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"0","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.109299974Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"1","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.109717676Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"2","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.109886011Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.109973167Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"[0]R flush_numbers[0..12]: [115, 118, 116, 116, 112, 121, 121, 109, 103, 113, 123, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110249446Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"downstairs","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"[0]R generation[0..12]: [64, 66, 65, 65, 64, 66, 66, 62, 60, 64, 67, 65]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110268045Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"downstairs","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110282396Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"downstairs","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"[1]R flush_numbers[0..12]: [115, 118, 116, 116, 112, 121, 121, 109, 103, 113, 123, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110296147Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"downstairs","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"[1]R generation[0..12]: [64, 66, 65, 65, 64, 66, 66, 62, 60, 64, 67, 65]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110309299Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"downstairs","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110326679Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"downstairs","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"[2]R flush_numbers[0..12]: [115, 118, 116, 116, 112, 121, 121, 109, 103, 113, 123, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.11034086Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"downstairs","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"[2]R generation[0..12]: [64, 66, 65, 65, 64, 66, 66, 62, 60, 64, 67, 65]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110354302Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"downstairs","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110367773Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"downstairs","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"Max found gen is 68","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:39:12.110380225Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"downstairs","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"Generation requested: 68 >= found:68","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110399733Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"downstairs","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"Next flush: 124","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110412235Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"downstairs","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110424508Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"":"downstairs","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.11043741Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110449113Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"0","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110461925Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"1","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110474807Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"client":"2","":"downstairs","session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110487539Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110499282Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"d2a24cf7-6912-4aa0-88ea-a5c352ace24e is now active with session: 449ccb99-f8af-4f66-b193-b18f1f57d6e8","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
,"time":"2025-01-27T19:39:12.110511844Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110525565Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454,"session_id":"449ccb99-f8af-4f66-b193-b18f1f57d6e8"}
{"msg":"The guest has finished waiting for activation with:68","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:12.110537538Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Flush
02/30 Write block 247 len 16384 data: 5 6 5 3
03/30 Read block 570 len 36864
04/30 Read block 31 len 16384
05/30 Read block 92 len 8192
06/30 Read block 328 len 20480
07/30 Read block 238 len 32768
08/30 Read block 204 len 4096
09/30 Write block 137 len 40960 data: 6 6 6 6 5 6 6 6 8 9
10/30 Read block 224 len 16384
11/30 Write block 305 len 16384 data: 7 8 10 10
12/30 Write block 244 len 12288 data: 6 6 5
13/30 Flush
14/30 Write block 359 len 4096 data: 10
15/30 Write block 279 len 24576 data: 8 7 6 5 2 2
16/30 Flush
17/30 Write block 89 len 32768 data: 6 8 7 10 10 8 8 7
18/30 Write block 59 len 16384 data: 5 5 6 7
19/30 Read block 327 len 40960
20/30 Write block 487 len 24576 data: 8 8 7 6 7 9
21/30 Write block 197 len 8192 data: 9 9
22/30 Read block 287 len 40960
23/30 Read block 531 len 12288
24/30 Read block 59 len 8192
25/30 Read block 529 len 32768
26/30 Read block 515 len 20480
27/30 Read block 305 len 20480
28/30 Write block 85 len 4096 data: 11
29/30 Read block 409 len 40960
30/30 Write block 548 len 28672 data: 5 7 6 8 8 6 8
----------------------------------------------------------------
Crucible gen:68 GIO:true work queues: Upstairs:0 downstairs:14
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1016 Acked Write 8 Done Done Done false
1017 Acked Write 4 Done Done Done false
1018 Acked Read 10 Done Done Done false
1019 Acked Write 6 Done Done Done false
1020 Acked Write 2 Done Done Done false
1021 Acked Read 10 Done Done Done false
1022 Acked Read 3 Done Done Done false
1023 Acked Read 2 Done Done Done false
1024 Acked Read 8 Done Done Done false
1025 Acked Read 5 Done Done Done false
1026 Acked Read 5 Done Done Done false
1027 Acked Write 1 Done Done Done false
1028 Acked Read 10 Done Done Done false
1029 Acked Write 7 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 13 13 13 39
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1015 1015 1015
Downstairs last five completed: 1015 1014 1013 1012 1011
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:12.129946389Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1454}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 64 64 64 115 115 115 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 65 65 65 116 116 116 T T T
3 060-079 65 65 65 116 116 116 T T T
4 080-099 64 64 64 112 112 112 T T T
5 100-119 66 66 66 121 121 121 F F F
6 120-139 68 68 68 125 125 125 F F F
7 140-159 68 68 68 125 125 125 F F F
8 160-179 60 60 60 103 103 103 F F F
9 180-199 64 64 64 113 113 113 T T T
10 200-219 67 67 67 123 123 123 F F F
11 220-239 65 65 65 116 116 116 F F F
12 240-259 68 68 68 125 125 125 F F F
13 260-279 68 68 68 126 126 126 F F F
14 280-299 68 68 68 126 126 126 F F F
15 300-319 68 68 68 125 125 125 F F F
16 320-339 66 66 66 120 120 120 F F F
17 340-359 68 68 68 126 126 126 F F F
18 360-379 60 60 60 104 104 104 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 67 67 67 123 123 123 F F F
21 420-439 66 66 66 122 122 122 F F F
22 440-459 64 64 64 115 115 115 F F F
23 460-479 64 64 64 115 115 115 F F F
24 480-499 59 59 59 101 101 101 T T T
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 65 65 65 116 116 116 T T T
28 560-579 62 62 62 109 109 109 F F F
29 580-599 62 62 62 109 109 109 F F F
Max gen: 68, Max flush: 126
A Difference in extent metadata is expected here
On loop 34
Verifying data now
No change after verify
Loop: 34 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 64 64 64 115 115 115 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 69 69 69 127 127 127 F F F
3 060-079 69 69 69 127 127 127 F F F
4 080-099 69 69 69 127 127 127 F F F
5 100-119 66 66 66 121 121 121 F F F
6 120-139 68 68 68 125 125 125 F F F
7 140-159 68 68 68 125 125 125 F F F
8 160-179 60 60 60 103 103 103 F F F
9 180-199 69 69 69 127 127 127 F F F
10 200-219 67 67 67 123 123 123 F F F
11 220-239 65 65 65 116 116 116 F F F
12 240-259 68 68 68 125 125 125 F F F
13 260-279 68 68 68 126 126 126 F F F
14 280-299 68 68 68 126 126 126 F F F
15 300-319 68 68 68 125 125 125 F F F
16 320-339 66 66 66 120 120 120 F F F
17 340-359 68 68 68 126 126 126 F F F
18 360-379 60 60 60 104 104 104 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 67 67 67 123 123 123 F F F
21 420-439 66 66 66 122 122 122 F F F
22 440-459 64 64 64 115 115 115 F F F
23 460-479 64 64 64 115 115 115 F F F
24 480-499 69 69 69 127 127 127 F F F
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 69 69 69 127 127 127 F F F
28 560-579 62 62 62 109 109 109 F F F
29 580-599 62 62 62 109 109 109 F F F
Max gen: 69, Max flush: 127
Begin loop 35
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.391194369Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.437993649Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.439179874Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.439223477Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.439243815Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.439259095Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462}
{"msg":"Crucible d9ed91bf-0c0f-400d-8980-92bdb05d23e5 has session id: f9282a57-70a4-4ebb-99f0-4dede2fbd923","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.439884108Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"Upstairs opts: Upstairs UUID: d9ed91bf-0c0f-400d-8980-92bdb05d23e5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.43992921Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"Crucible stats registered with UUID: d9ed91bf-0c0f-400d-8980-92bdb05d23e5","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.439959781Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:16.439982597Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.440385797Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"task":"crutest"}
{"msg":"The guest has requested activation with gen:70","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.440419027Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462}
{"msg":"d9ed91bf-0c0f-400d-8980-92bdb05d23e5 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.44047847Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.440506313Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"0","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.44054243Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"0","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.440571242Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"1","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.440653941Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"1","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.440689629Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"2","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.440728145Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"2","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.440760096Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"io task","client":"2","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.440787928Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"io task","client":"0","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.440816661Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"io task","client":"1","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"ds_connection connected from Ok(127.0.0.1:44174)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.440846052Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"io task","client":"2","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"ds_connection connected from Ok(127.0.0.1:51520)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.440874265Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"io task","client":"0","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49627)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.440903517Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"io task","client":"1","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.441617554Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"1","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.441658809Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"0","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.441871917Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"2","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.442423346Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"1","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.442451288Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"0","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.442471186Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"2","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.442996541Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443097368Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"[0]R flush_numbers[0..12]: [115, 118, 127, 127, 127, 121, 125, 125, 103, 127, 123, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443231195Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"downstairs","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"[0]R generation[0..12]: [64, 66, 69, 69, 69, 66, 68, 68, 60, 69, 67, 65]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443251093Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"downstairs","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443265744Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"downstairs","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"[1]R flush_numbers[0..12]: [115, 118, 127, 127, 127, 121, 125, 125, 103, 127, 123, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443279205Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"downstairs","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"[1]R generation[0..12]: [64, 66, 69, 69, 69, 66, 68, 68, 60, 69, 67, 65]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443292417Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"downstairs","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443306998Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"downstairs","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"[2]R flush_numbers[0..12]: [115, 118, 127, 127, 127, 121, 125, 125, 103, 127, 123, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.44332038Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"downstairs","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"[2]R generation[0..12]: [64, 66, 69, 69, 69, 66, 68, 68, 60, 69, 67, 65]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443333682Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"downstairs","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443347303Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"downstairs","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"Max found gen is 70","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443359726Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"downstairs","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"Generation requested: 70 >= found:70","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443372548Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"downstairs","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"Next flush: 128","v":0,"name":"crucible","level"Wait for a query_work_queue command to finish before sending IO
:30,"time":"2025-01-27T19:39:16.443391566Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"downstairs","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443404248Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"":"downstairs","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.44341721Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443431391Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"0","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443444523Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"1","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443457165Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"client":"2","":"downstairs","session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443469847Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.44348177Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"d9ed91bf-0c0f-400d-8980-92bdb05d23e5 is now active with session: f9282a57-70a4-4ebb-99f0-4dede2fbd923","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443494263Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443505815Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462,"session_id":"f9282a57-70a4-4ebb-99f0-4dede2fbd923"}
{"msg":"The guest has finished waiting for activation with:70","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:16.443517558Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 247 len 4096 data: 6
02/30 Read block 515 len 28672
03/30 Write block 119 len 24576 data: 10 8 8 7 4 5
04/30 Write block 149 len 40960 data: 8 8 7 7 11 10 6 7 5 5
05/30 Read block 157 len 12288
06/30 Write block 226 len 16384 data: 2 2 3 2
07/30 Read block 517 len 4096
08/30 Read block 531 len 12288
09/30 Read block 53 len 20480
10/30 Read block 365 len 24576
11/30 Read block 53 len 12288
12/30 Read block 537 len 16384
13/30 Read block 458 len 20480
14/30 Write block 565 len 12288 data: 6 4 4
15/30 Read block 124 len 28672
16/30 Read block 273 len 8192
17/30 Flush
18/30 Write block 418 len 40960 data: 5 6 6 4 4 6 6 6 5 5
19/30 Read block 505 len 24576
20/30 Read block 368 len 32768
21/30 Write block 467 len 40960 data: 5 5 5 5 4 4 4 5 5 4
22/30 Write block 455 len 20480 data: 9 7 6 5 5
23/30 Read block 489 len 40960
24/30 Read block 429 len 24576
25/30 Write block 104 len 12288 data: 10 10 9
26/30 Flush
27/30 Write block 281 len 28672 data: 7 6 3 3 3 3 3
28/30 Read block 479 len 40960
29/30 Read block 408 len 16384
30/30 Write block 264 len 40960 data: 7 8 7 5 6 9 8 9 8 8
----------------------------------------------------------------
Crucible gen:70 GIO:true work queues: Upstairs:0 downstairs:4
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1026 Acked Write 7 Done Done Done false
1027 Acked Read 10 Done Done Done false
1028 Acked Read 4 Done Done Done false
1029 Acked Write 10 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 3 3 3 9
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1025 1025 1025
Downstairs last five completed: 1025 1024 1023 1022 1021
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:16.464802507Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1462}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 64 64 64 115 115 115 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 69 69 69 127 127 127 F F F
3 060-079 69 69 69 127 127 127 F F F
4 080-099 69 69 69 127 127 127 F F F
5 100-119 70 70 70 129 129 129 F F F
6 120-139 70 70 70 128 128 128 F F F
7 140-159 70 70 70 128 128 128 F F F
8 160-179 60 60 60 103 103 103 F F F
9 180-199 69 69 69 127 127 127 F F F
10 200-219 67 67 67 123 123 123 F F F
11 220-239 70 70 70 128 128 128 F F F
12 240-259 70 70 70 128 128 128 F F F
13 260-279 68 68 68 126 126 126 T T T
14 280-299 68 68 68 126 126 126 T T T
15 300-319 68 68 68 125 125 125 F F F
16 320-339 66 66 66 120 120 120 F F F
17 340-359 68 68 68 126 126 126 F F F
18 360-379 60 60 60 104 104 104 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 70 70 70 129 129 129 F F F
21 420-439 70 70 70 129 129 129 F F F
22 440-459 70 70 70 129 129 129 F F F
23 460-479 70 70 70 129 129 129 F F F
24 480-499 69 69 69 127 127 127 F F F
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 69 69 69 127 127 127 F F F
28 560-579 70 70 70 128 128 128 F F F
29 580-599 62 62 62 109 109 109 F F F
Max gen: 70, Max flush: 129
A Difference in extent metadata is expected here
On loop 35
Verifying data now
No change after verify
Loop: 35 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 64 64 64 115 115 115 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 69 69 69 127 127 127 F F F
3 060-079 69 69 69 127 127 127 F F F
4 080-099 69 69 69 127 127 127 F F F
5 100-119 70 70 70 129 129 129 F F F
6 120-139 70 70 70 128 128 128 F F F
7 140-159 70 70 70 128 128 128 F F F
8 160-179 60 60 60 103 103 103 F F F
9 180-199 69 69 69 127 127 127 F F F
10 200-219 67 67 67 123 123 123 F F F
11 220-239 70 70 70 128 128 128 F F F
12 240-259 70 70 70 128 128 128 F F F
13 260-279 71 71 71 130 130 130 F F F
14 280-299 71 71 71 130 130 130 F F F
15 300-319 68 68 68 125 125 125 F F F
16 320-339 66 66 66 120 120 120 F F F
17 340-359 68 68 68 126 126 126 F F F
18 360-379 60 60 60 104 104 104 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 70 70 70 129 129 129 F F F
21 420-439 70 70 70 129 129 129 F F F
22 440-459 70 70 70 129 129 129 F F F
23 460-479 70 70 70 129 129 129 F F F
24 480-499 69 69 69 127 127 127 F F F
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 69 69 69 127 127 127 F F F
28 560-579 70 70 70 128 128 128 F F F
29 580-599 62 62 62 109 109 109 F F F
Max gen: 71, Max flush: 130
Begin loop 36
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.706187368Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:20.750926343Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.751590762Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.784527236Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.785468213Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.785504101Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.785517883Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.785531714Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470}
{"msg":"Crucible 7f7c8a53-b4be-47d6-b06b-29a2ee696d83 has session id: f3ea0880-9247-4ae0-90ec-730605b6f999","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786129334Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"Upstairs opts: Upstairs UUID: 7f7c8a53-b4be-47d6-b06b-29a2ee696d83, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786165571Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"Crucible stats registered with UUID: 7f7c8a53-b4be-47d6-b06b-29a2ee696d83","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786182461Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:20.786195043Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786615533Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"task":"crutest"}
{"msg":"The guest has requested activation with gen:72","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786639518Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470}
{"msg":"7f7c8a53-b4be-47d6-b06b-29a2ee696d83 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786729812Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786747941Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"0","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786772495Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"0","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786786327Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"1","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786800248Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"1","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786822504Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"2","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786849398Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"2","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786875072Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"io task","client":"2","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786900746Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"io task","client":"0","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786923842Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"io task","client":"1","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"ds_connection connected from Ok(127.0.0.1:46325)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786947847Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"io task","client":"2","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"ds_connection connected from Ok(127.0.0.1:34293)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.786972602Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"io task","client":"0","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"ds_connection connected from Ok(127.0.0.1:63500)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.787126247Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"io task","client":"1","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.787667212Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"2","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.787707926Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"1","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.78799222Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"0","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.788470964Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"1","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.788512968Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"2","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.788547526Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"0","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789068364Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789095077Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"[0]R flush_numbers[0..12]: [115, 118, 127, 127, 127, 129, 128, 128, 103, 127, 123, 128]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789123819Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"downstairs","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"[0]R generation[0..12]: [64, 66, 69, 69, 69, 70, 70, 70, 60, 69, 67, 70]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.78913927Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"downstairs","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789153691Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"downstairs","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"[1]R flush_numbers[0..12]: [115, 118, 127, 127, 127, 129, 128, 128, 103, 127, 123, 128]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789167452Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"downstairs","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"[1]R generation[0..12]: [64, 66, 69, 69, 69, 70, 70, 70, 60, 69, 67, 70]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789180614Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"downstairs","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789194146Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"downstairs","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"[2]R flush_numbers[0..12]: [115, 118, 127, 127, 127, 129, 128, 128, 103, 127, 123, 128]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789207627Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"downstairs","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"[2]R generation[0..12]: [64, 66, 69, 69, 69, 70, 70, 70, 60, 69, 67, 70]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789220819Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"downstairs","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789237959Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"downstairs","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"Max found gen is 72","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789250721Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"downstairs","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"Generation requested: 72 >= found:72","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789263563Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"downstairs","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"Next flush: 131","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789275945Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"downstairs","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
Wait for a query_work_queue command to finish before sending IO
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789294164Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"":"downstairs","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789306996Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789318449Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"0","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789331261Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"1","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789343883Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"client":"2","":"downstairs","session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789356495Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789368408Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"7f7c8a53-b4be-47d6-b06b-29a2ee696d83 is now active with session: f3ea0880-9247-4ae0-90ec-730605b6f999","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789380651Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789392203Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470,"session_id":"f3ea0880-9247-4ae0-90ec-730605b6f999"}
{"msg":"The guest has finished waiting for activation with:72","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:20.789404096Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 301 len 40960 data: 10 7 9 7 8 9 11 11 10 9
02/30 Read block 92 len 16384
03/30 Read block 21 len 12288
04/30 Read block 312 len 16384
05/30 Write block 246 len 40960 data: 6 7 7 6 4 4 3 3 5 6
06/30 Read block 50 len 24576
07/30 Write block 348 len 20480 data: 6 7 6 5 5
08/30 Read block 364 len 4096
09/30 Flush
10/30 Write block 458 len 20480 data: 6 6 6 5 6
11/30 Write block 270 len 24576 data: 9 10 9 9 11 11
12/30 Write block 467 len 28672 data: 6 6 6 6 5 5 5
13/30 Read block 117 len 8192
14/30 Read block 574 len 32768
15/30 Read block 378 len 8192
16/30 Flush
17/30 Write block 153 len 8192 data: 12 11
18/30 Read block 157 len 8192
19/30 Write block 428 len 36864 data: 5 6 7 6 7 7 9 10 7
20/30 Write block 181 len 12288 data: 11 10 8
21/30 Read block 592 len 28672
22/30 Read block 255 len 36864
23/30 Write block 9 len 24576 data: 7 7 6 4 3 3
24/30 Write block 161 len 28672 data: 7 6 5 8 8 9 8
25/30 Write block 365 len 20480 data: 3 3 4 5 7
26/30 Write block 70 len 24576 data: 8 8 6 7 7 6
27/30 Read block 538 len 28672
28/30 Write block 86 len 12288 data: 10 7 7
29/30 Write block 361 len 8192 data: 6 5
30/30 Write block 487 len 4096 data: 9
----------------------------------------------------------------
Crucible gen:72 GIO:true work queues: Upstairs:0 downstairs:14
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1016 Acked Write 2 Done Done Done false
1017 Acked Read 2 Done Done Done false
1018 Acked Write 9 Done Done Done false
1019 Acked Write 3 Done Done Done false
1020 Acked Read 7 Done Done Done false
1021 Acked Read 9 Done Done Done false
1022 Acked Write 6 Done Done Done false
1023 Acked Write 7 Done Done Done false
1024 Acked Write 5 Done Done Done false
1025 Acked Write 6 Done Done Done false
1026 Acked Read 7 Done Done Done false
1027 Acked Write 3 Done Sent Sent false
1028 Acked Write 2 Sent Sent Sent false
1029 Acked Write 1 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 3 3 8
Done 12 11 11 34
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1015 1015 1015
Downstairs last five completed: 1015 1014 1013 1012 1011
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:20.808989688Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1470}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 64 64 64 115 115 115 T T T
1 020-039 66 66 66 118 118 118 F F F
2 040-059 69 69 69 127 127 127 F F F
3 060-079 69 69 69 127 127 127 T T T
4 080-099 69 69 69 127 127 127 T T T
5 100-119 70 70 70 129 129 129 F F F
6 120-139 70 70 70 128 128 128 F F F
7 140-159 70 70 70 128 128 128 T T T
8 160-179 60 60 60 103 103 103 T T T
9 180-199 69 69 69 127 127 127 T T T
10 200-219 67 67 67 123 123 123 F F F
11 220-239 70 70 70 128 128 128 F F F
12 240-259 72 72 72 131 131 131 F F F
13 260-279 72 72 72 132 132 132 F F F
14 280-299 71 71 71 130 130 130 F F F
15 300-319 72 72 72 131 131 131 F F F
16 320-339 66 66 66 120 120 120 F F F
17 340-359 72 72 72 131 131 131 F F F
18 360-379 60 60 60 104 104 104 T T T
19 380-399 60 60 60 104 104 104 F F F
20 400-419 70 70 70 129 129 129 F F F
21 420-439 70 70 70 129 129 129 T T T
22 440-459 72 72 72 132 132 132 F F F
23 460-479 72 72 72 132 132 132 F F F
24 480-499 69 69 69 127 127 127 T T T
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 69 69 69 127 127 127 F F F
28 560-579 70 70 70 128 128 128 F F F
29 580-599 62 62 62 109 109 109 F F F
Max gen: 72, Max flush: 132
A Difference in extent metadata is expected here
On loop 36
Verifying data now
No change after verify
Loop: 36 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 69 69 69 127 127 127 F F F
3 060-079 73 73 73 133 133 133 F F F
4 080-099 73 73 73 133 133 133 F F F
5 100-119 70 70 70 129 129 129 F F F
6 120-139 70 70 70 128 128 128 F F F
7 140-159 73 73 73 133 133 133 F F F
8 160-179 73 73 73 133 133 133 F F F
9 180-199 73 73 73 133 133 133 F F F
10 200-219 67 67 67 123 123 123 F F F
11 220-239 70 70 70 128 128 128 F F F
12 240-259 72 72 72 131 131 131 F F F
13 260-279 72 72 72 132 132 132 F F F
14 280-299 71 71 71 130 130 130 F F F
15 300-319 72 72 72 131 131 131 F F F
16 320-339 66 66 66 120 120 120 F F F
17 340-359 72 72 72 131 131 131 F F F
18 360-379 73 73 73 133 133 133 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 70 70 70 129 129 129 F F F
21 420-439 73 73 73 133 133 133 F F F
22 440-459 72 72 72 132 132 132 F F F
23 460-479 72 72 72 132 132 132 F F F
24 480-499 73 73 73 133 133 133 F F F
25 500-519 58 58 58 99 99 99 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 69 69 69 127 127 127 F F F
28 560-579 70 70 70 128 128 128 F F F
29 580-599 62 62 62 109 109 109 F F F
Max gen: 73, Max flush: 133
Begin loop 37
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.115632338Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.160220327Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.161008628Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.161060216Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.161078365Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.161096394Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478}
{"msg":"Crucible d780e794-6b6b-4c0c-af29-a1e0d12136c6 has session id: e6eca0ba-73a8-41f0-8c9b-e9429f13f27d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.161664962Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"Upstairs opts: Upstairs UUID: d780e794-6b6b-4c0c-af29-a1e0d12136c6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.161718699Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"Crucible stats registered with UUID: d780e794-6b6b-4c0c-af29-a1e0d12136c6","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.1617495Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:25.161776433Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162160195Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"task":"crutest"}
{"msg":"The guest has requested activation with gen:74","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162182991Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478}
{"msg":"d780e794-6b6b-4c0c-af29-a1e0d12136c6 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162251249Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162268858Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"0","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162293023Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"0","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162306475Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"1","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162320776Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"1","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162333798Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"2","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162347609Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"2","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162360641Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"io task","client":"2","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162380799Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"io task","client":"0","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162404734Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"io task","client":"1","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:36985)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162430318Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"io task","client":"2","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47144)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162454283Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"io task","client":"0","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50555)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.162678345Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"io task","client":"1","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.163139439Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"0","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.163176246Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"1","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.163823875Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"1","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.163863651Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"0","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.164516136Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.16454297Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"2","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.164564337Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165079098Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"2","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"[0]R flush_numbers[0..12]: [133, 118, 127, 133, 133, 129, 128, 133, 133, 133, 123, 128]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165583436Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"downstairs","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"[0]R generation[0..12]: [73, 66, 69, 73, 73, 70, 70, 73, 73, 73, 67, 70]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165605962Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"downstairs","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165624001Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"downstairs","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"[1]R flush_numbers[0..12]: [133, 118, 127, 133, 133, 129, 128, 133, 133, 133, 123, 128]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165637792Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"downstairs","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"[1]R generation[0..12]: [73, 66, 69, 73, 73, 70, 70, 73, 73, 73, 67, 70]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165651054Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"downstairs","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165664675Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"downstairs","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"[2]R flush_numbers[0..12]: [133, 118, 127, 133, 133, 129, 128, 133, 133, 133, 123, 128]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165678007Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"downstairs","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"[2]R generation[0..12]: [73, 66, 69, 73, 73, 70, 70, 73, 73, 73, 67, 70]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165691169Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"downstairs","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165704611Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"downstairs","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"Max found gen is 74","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165717003Z"Wait for a query_work_queue command to finish before sending IO
,"hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"downstairs","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"Generation requested: 74 >= found:74","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165736421Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"downstairs","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"Next flush: 134","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165749033Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"downstairs","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165761635Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"":"downstairs","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165774297Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.16578604Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"0","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165799002Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"1","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165813203Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"client":"2","":"downstairs","session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165826125Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165837968Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"d780e794-6b6b-4c0c-af29-a1e0d12136c6 is now active with session: e6eca0ba-73a8-41f0-8c9b-e9429f13f27d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.16585047Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165862073Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478,"session_id":"e6eca0ba-73a8-41f0-8c9b-e9429f13f27d"}
{"msg":"The guest has finished waiting for activation with:74","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:25.165873956Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 191 len 36864 data: 7 6 7 7 10 8 10 10 7
02/30 Read block 84 len 12288
03/30 Read block 159 len 28672
04/30 Flush
05/30 Write block 507 len 36864 data: 4 5 5 5 6 5 3 3 4
06/30 Flush
07/30 Write block 552 len 4096 data: 9
08/30 Read block 349 len 40960
09/30 Read block 469 len 28672
10/30 Write block 432 len 36864 data: 8 8 10 11 8 4 4 3 4
11/30 Write block 593 len 20480 data: 3 3 3 4 4
12/30 Read block 328 len 24576
13/30 Read block 344 len 40960
14/30 Read block 271 len 20480
15/30 Read block 534 len 20480
16/30 Write block 400 len 24576 data: 4 4 5 6 6 6
17/30 Read block 549 len 24576
18/30 Write block 143 len 28672 data: 7 7 9 10 8 8 9
19/30 Write block 41 len 4096 data: 9
20/30 Read block 360 len 40960
21/30 Read block 209 len 12288
22/30 Read block 444 len 8192
23/30 Read block 73 len 40960
24/30 Read block 278 len 4096
25/30 Write block 179 len 36864 data: 11 10 12 11 9 6 4 5 7
26/30 Read block 507 len 24576
27/30 Write block 246 len 4096 data: 7
28/30 Read block 365 len 8192
29/30 Write block 341 len 8192 data: 7 7
30/30 Read block 43 len 28672
----------------------------------------------------------------
Crucible gen:74 GIO:true work queues: Upstairs:0 downstairs:24
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1006 Acked Write 1 Done Done Done false
1007 Acked Read 10 Done Done Done false
1008 Acked Read 7 Done Done Done false
1009 Acked Write 9 Done Done Done false
1010 Acked Write 5 Done Done Done false
1011 Acked Read 6 Done Done Done false
1012 Acked Read 10 Done Done Done false
1013 Acked Read 5 Done Done Done false
1014 Acked Read 5 Done Done Done false
1015 Acked Write 6 Done Done Done false
1016 Acked Read 6 Done Done Done false
1017 Acked Write 7 Done Done Done false
1018 Acked Write 1 Done Done Done false
1019 Acked Read 10 Done Done Done false
1020 Acked Read 3 Done Done Done false
1021 Acked Read 2 Done Done Done false
1022 Acked Read 10 Done Done Done false
1023 Acked Read 1 Done Done Done false
1024 Acked Write 9 Done Done Done false
1025 Acked Read 6 Done Done Done false
1026 Acked Write 1 Done Done Done false
1027 Acked Read 2 Done Done Done false
1028 Acked Write 2 Done Done Done false
1029 Acked Read 7 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 1 1
Done 24 24 23 71
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1005 1005 1005
Downstairs last five completed: 1005 1004 1003 1002 1001
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:25.188149981Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1478}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 69 69 69 127 127 127 T T T
3 060-079 73 73 73 133 133 133 F F F
4 080-099 73 73 73 133 133 133 F F F
5 100-119 70 70 70 129 129 129 F F F
6 120-139 70 70 70 128 128 128 F F F
7 140-159 73 73 73 133 133 133 T T T
8 160-179 73 73 73 133 133 133 T T T
9 180-199 74 74 74 134 134 134 T T T
10 200-219 67 67 67 123 123 123 F F F
11 220-239 70 70 70 128 128 128 F F F
12 240-259 72 72 72 131 131 131 T T T
13 260-279 72 72 72 132 132 132 F F F
14 280-299 71 71 71 130 130 130 F F F
15 300-319 72 72 72 131 131 131 F F F
16 320-339 66 66 66 120 120 120 F F F
17 340-359 72 72 72 131 131 131 T T T
18 360-379 73 73 73 133 133 133 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 70 70 70 129 129 129 T T T
21 420-439 73 73 73 133 133 133 T T T
22 440-459 72 72 72 132 132 132 T T T
23 460-479 72 72 72 132 132 132 F F F
24 480-499 73 73 73 133 133 133 F F F
25 500-519 74 74 74 135 135 135 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 69 69 69 127 127 127 T T T
28 560-579 70 70 70 128 128 128 F F F
29 580-599 62 62 62 109 109 109 T T T
Max gen: 74, Max flush: 135
A Difference in extent metadata is expected here
On loop 37
Verifying data now
No change after verify
Loop: 37 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 75 75 75 136 136 136 F F F
3 060-079 73 73 73 133 133 133 F F F
4 080-099 73 73 73 133 133 133 F F F
5 100-119 70 70 70 129 129 129 F F F
6 120-139 70 70 70 128 128 128 F F F
7 140-159 75 75 75 136 136 136 F F F
8 160-179 75 75 75 136 136 136 F F F
9 180-199 75 75 75 136 136 136 F F F
10 200-219 67 67 67 123 123 123 F F F
11 220-239 70 70 70 128 128 128 F F F
12 240-259 75 75 75 136 136 136 F F F
13 260-279 72 72 72 132 132 132 F F F
14 280-299 71 71 71 130 130 130 F F F
15 300-319 72 72 72 131 131 131 F F F
16 320-339 66 66 66 120 120 120 F F F
17 340-359 75 75 75 136 136 136 F F F
18 360-379 73 73 73 133 133 133 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 75 75 75 136 136 136 F F F
21 420-439 75 75 75 136 136 136 F F F
22 440-459 75 75 75 136 136 136 F F F
23 460-479 72 72 72 132 132 132 F F F
24 480-499 73 73 73 133 133 133 F F F
25 500-519 74 74 74 135 135 135 F F F
26 520-539 58 58 58 100 100 100 F F F
27 540-559 75 75 75 136 136 136 F F F
28 560-579 70 70 70 128 128 128 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 75, Max flush: 136
Begin loop 38
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.477461644Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.52210302Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.523192615Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.523235089Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.52324946Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.523263851Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486}
{"msg":"Crucible ba8ea41f-f082-4795-83e1-aa99ab172b1e has session id: cb8266f2-f586-438c-b219-979a075edb9a","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.523902306Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"Upstairs opts: Upstairs UUID: ba8ea41f-f082-4795-83e1-aa99ab172b1e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.523934326Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"Crucible stats registered with UUID: ba8ea41f-f082-4795-83e1-aa99ab172b1e","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.523948667Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:29.523960809Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524455443Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"task":"crutest"}
{"msg":"The guest has requested activation with gen:76","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524481057Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486}
{"msg":"ba8ea41f-f082-4795-83e1-aa99ab172b1e active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524547206Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524575139Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"0","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524611826Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"0","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524634033Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"1","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524657428Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"1","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524679934Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"2","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524706388Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"2","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524745434Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"io task","client":"2","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524772457Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"io task","client":"0","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.52479988Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"io task","client":"1","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"ds_connection connected from Ok(127.0.0.1:54267)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524825584Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"io task","client":"1","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47792)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524850978Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"io task","client":"0","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"ds_connection connected from Ok(127.0.0.1:44258)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.524875863Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"io task","client":"2","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.525622411Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"2","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.525662786Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"0","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.525696315Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"1","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.52628524Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"0","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.526313213Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"2","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.526369248Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"1","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.526892894Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.526918248Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"[0]R flush_numbers[0..12]: [133, 118, 136, 133, 133, 129, 128, 136, 136, 136, 123, 128]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.526975662Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"downstairs","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"[0]R generation[0..12]: [73, 66, 75, 73, 73, 70, 70, 75, 75, 75, 67, 70]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.526995021Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"downstairs","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527009871Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"downstairs","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"[1]R flush_numbers[0..12]: [133, 118, 136, 133, 133, 129, 128, 136, 136, 136, 123, 128]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527023803Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"downstairs","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"[1]R generation[0..12]: [73, 66, 75, 73, 73, 70, 70, 75, 75, 75, 67, 70]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527037394Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"downstairs","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527051146Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"downstairs","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"[2]R flush_numbers[0..12]: [133, 118, 136, 133, 133, 129, 128, 136, 136, 136, 123, 128]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527064498Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"downstairs","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"[2]R generation[0..12]: [73, 66, 75, 73, 73, 70, 70, 75, 75, 75, 67, 70]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.52707745Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"downstairs","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527094869Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"downstairs","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"Max found gen is 76","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527107751Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"downstairs","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"Generation requested: 76 >= found:76","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527120633Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"downstairs","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"Next flush: 137","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527133205Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"downstairs","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527145707Z","hostname":"Wait for a query_work_queue command to finish before sending IO
w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"":"downstairs","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.52717382Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527198025Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"0","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527223919Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"1","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527250153Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"client":"2","":"downstairs","session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527276656Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527301911Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"ba8ea41f-f082-4795-83e1-aa99ab172b1e is now active with session: cb8266f2-f586-438c-b219-979a075edb9a","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527327335Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
,"time":"2025-01-27T19:39:29.52735086Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486,"session_id":"cb8266f2-f586-438c-b219-979a075edb9a"}
{"msg":"The guest has finished waiting for activation with:76","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:29.527380562Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 247 len 12288 data: 8 8 7
02/30 Read block 438 len 8192
03/30 Read block 71 len 20480
04/30 Read block 250 len 12288
05/30 Write block 323 len 20480 data: 8 9 10 8 9
06/30 Write block 534 len 8192 data: 8 10
07/30 Read block 530 len 32768
08/30 Write block 528 len 24576 data: 11 9 10 10 9 9
09/30 Read block 49 len 32768
10/30 Write block 95 len 40960 data: 9 8 7 6 6 8 9 8 8 11
11/30 Flush
12/30 Write block 568 len 12288 data: 4 4 6
13/30 Read block 400 len 12288
14/30 Read block 392 len 28672
15/30 Flush
16/30 Write block 529 len 40960 data: 10 11 11 10 10 9 11 8 7 6
17/30 Write block 357 len 40960 data: 9 11 11 9 7 6 4 3 4 4
18/30 Read block 104 len 28672
19/30 Read block 428 len 8192
20/30 Read block 475 len 20480
21/30 Read block 498 len 36864
22/30 Write block 539 len 12288 data: 5 5 6
23/30 Read block 144 len 4096
24/30 Write block 256 len 24576 data: 6 7 4 3 6 6
25/30 Read block 135 len 20480
26/30 Read block 127 len 24576
27/30 Read block 456 len 40960
28/30 Write block 430 len 16384 data: 8 7 9 9
29/30 Read block 374 len 24576
30/30 Write block 181 len 24576 data: 13 12 10 7 5 6
----------------------------------------------------------------
Crucible gen:76 GIO:true work queues: Upstairs:0 downstairs:15
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1015 Acked Write 10 Done Done Done false
1016 Acked Write 10 Done Done Done false
1017 Acked Read 7 Done Done Done false
1018 Acked Read 2 Done Done Done false
1019 Acked Read 5 Done Done Done false
1020 Acked Read 9 Done Done Done false
1021 Acked Write 3 Done Done Done false
1022 Acked Read 1 Done Done Done false
1023 Acked Write 6 Done Done Done false
1024 Acked Read 5 Done Done Done false
1025 Acked Read 6 Done Done Done false
1026 Acked Read 10 Done Done Done false
1027 Acked Write 4 Done Done Done false
1028 Acked Read 6 Done Done Done false
1029 Acked Write 6 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 14 14 14 42
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1014 1014 1014
Downstairs last five completed: 1014 1013 1012 1011 1010
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:29.5486827Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1486}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 75 75 75 136 136 136 F F F
3 060-079 73 73 73 133 133 133 F F F
4 080-099 76 76 76 137 137 137 F F F
5 100-119 76 76 76 137 137 137 F F F
6 120-139 70 70 70 128 128 128 F F F
7 140-159 75 75 75 136 136 136 F F F
8 160-179 75 75 75 136 136 136 F F F
9 180-199 75 75 75 136 136 136 T T T
10 200-219 67 67 67 123 123 123 F F F
11 220-239 70 70 70 128 128 128 F F F
12 240-259 76 76 76 137 137 137 T T T
13 260-279 72 72 72 132 132 132 T T T
14 280-299 71 71 71 130 130 130 F F F
15 300-319 72 72 72 131 131 131 F F F
16 320-339 76 76 76 137 137 137 F F F
17 340-359 75 75 75 136 136 136 T T T
18 360-379 73 73 73 133 133 133 T T T
19 380-399 60 60 60 104 104 104 F F F
20 400-419 75 75 75 136 136 136 F F F
21 420-439 75 75 75 136 136 136 T T T
22 440-459 75 75 75 136 136 136 F F F
23 460-479 72 72 72 132 132 132 F F F
24 480-499 73 73 73 133 133 133 F F F
25 500-519 74 74 74 135 135 135 F F F
26 520-539 76 76 76 137 137 137 T T T
27 540-559 75 75 75 136 136 136 T T T
28 560-579 76 76 76 138 138 138 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 76, Max flush: 138
A Difference in extent metadata is expected here
On loop 38
Verifying data now
No change after verify
Loop: 38 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 75 75 75 136 136 136 F F F
3 060-079 73 73 73 133 133 133 F F F
4 080-099 76 76 76 137 137 137 F F F
5 100-119 76 76 76 137 137 137 F F F
6 120-139 70 70 70 128 128 128 F F F
7 140-159 75 75 75 136 136 136 F F F
8 160-179 75 75 75 136 136 136 F F F
9 180-199 77 77 77 139 139 139 F F F
10 200-219 67 67 67 123 123 123 F F F
11 220-239 70 70 70 128 128 128 F F F
12 240-259 77 77 77 139 139 139 F F F
13 260-279 77 77 77 139 139 139 F F F
14 280-299 71 71 71 130 130 130 F F F
15 300-319 72 72 72 131 131 131 F F F
16 320-339 76 76 76 137 137 137 F F F
17 340-359 77 77 77 139 139 139 F F F
18 360-379 77 77 77 139 139 139 F F F
19 380-399 60 60 60 104 104 104 F F F
20 400-419 75 75 75 136 136 136 F F F
21 420-439 77 77 77 139 139 139 F F F
22 440-459 75 75 75 136 136 136 F F F
23 460-479 72 72 72 132 132 132 F F F
24 480-499 73 73 73 133 133 133 F F F
25 500-519 74 74 74 135 135 135 F F F
26 520-539 77 77 77 139 139 139 F F F
27 540-559 77 77 77 139 139 139 F F F
28 560-579 76 76 76 138 138 138 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 77, Max flush: 139
Begin loop 39
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.820037942Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.865484619Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.866417181Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.866466531Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.86648466Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.866507705Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494}
{"msg":"Crucible 31ad1524-7fa7-47ac-8869-7f1b039cd18f has session id: 794ee026-823e-4032-b27d-b1c78e62032b","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867079311Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"Upstairs opts: Upstairs UUID: 31ad1524-7fa7-47ac-8869-7f1b039cd18f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867123464Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"Crucible stats registered with UUID: 31ad1524-7fa7-47ac-8869-7f1b039cd18f","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867150497Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:33.867171814Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867639325Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"task":"crutest"}
{"msg":"The guest has requested activation with gen:78","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867664339Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494}
{"msg":"31ad1524-7fa7-47ac-8869-7f1b039cd18f active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867723483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867741511Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"0","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867764897Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"0","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867779038Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"1","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867800245Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"1","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.86782414Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"2","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867852503Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"2","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867875389Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"io task","client":"2","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.86788965Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"io task","client":"0","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867906949Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"io task","client":"1","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"ds_connection connected from Ok(127.0.0.1:58435)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.867961185Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"io task","client":"2","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52062)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.868217037Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"io task","client":"1","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"ds_connection connected from Ok(127.0.0.1:56666)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.868257722Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"io task","client":"0","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.868819894Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"0","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.868854492Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"2","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.869238844Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"1","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.869459248Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"0","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.869613522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"2","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.869998164Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"1","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.87006919Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870095874Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"[0]R flush_numbers[0..12]: [133, 118, 136, 133, 137, 137, 128, 136, 136, 139, 123, 128]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870632171Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"downstairs","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"[0]R generation[0..12]: [73, 66, 75, 73, 76, 76, 70, 75, 75, 77, 67, 70]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870651589Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"downstairs","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870669878Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"downstairs","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"[1]R flush_numbers[0..12]: [133, 118, 136, 133, 137, 137, 128, 136, 136, 139, 123, 128]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870684159Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"downstairs","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"[1]R generation[0..12]: [73, 66, 75, 73, 76, 76, 70, 75, 75, 77, 67, 70]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870697361Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"downstairs","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870710943Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"downstairs","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"[2]R flush_numbers[0..12]: [133, 118, 136, 133, 137, 137, 128, 136, 136, 139, 123, 128]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870725234Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"downstairs","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"[2]R generation[0..12]: [73, 66, 75, 73, 76, 76, 70, 75, 75, 77, 67, 70]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870738485Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"downstairs","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870751937Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"downstairs","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"Max found gen is 78","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:39:33.87076432Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"downstairs","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"Generation requested: 78 >= found:78","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870782138Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"downstairs","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"Next flush: 140","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870794611Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"downstairs","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870806973Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"":"downstairs","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870819535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870831148Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"0","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.87084401Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"1","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870856622Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"client":"2","":"downstairs","session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870871003Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870882826Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"31ad1524-7fa7-47ac-8869-7f1b039cd18f is now active with session: 794ee026-823e-4032-b27d-b1c78e62032b","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870895248Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870906911Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494,"session_id":"794ee026-823e-4032-b27d-b1c78e62032b"}
{"msg":"The guest has finished waiting for activation with:78","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:33.870918864Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 215 len 40960 data: 6 5 5 6 5 5 4 4 4 3
02/30 Read block 155 len 40960
03/30 Write block 80 len 8192 data: 6 7
04/30 Flush
05/30 Write block 490 len 24576 data: 7 8 10 8 5 9
06/30 Read block 67 len 12288
07/30 Write block 396 len 16384 data: 5 4 3 2
08/30 Read block 186 len 36864
09/30 Read block 465 len 32768
10/30 Write block 368 len 36864 data: 6 8 9 9 7 8 10 11 11
11/30 Write block 234 len 16384 data: 3 3 2 3
12/30 Read block 549 len 8192
13/30 Read block 409 len 36864
14/30 Read block 150 len 4096
15/30 Flush
16/30 Write block 543 len 36864 data: 4 4 3 5 6 6 8 7 9
17/30 Read block 253 len 28672
18/30 Write block 304 len 12288 data: 8 9 10
19/30 Read block 533 len 8192
20/30 Write block 179 len 12288 data: 12 11 14
21/30 Read block 591 len 12288
22/30 Write block 247 len 8192 data: 9 9
23/30 Write block 502 len 28672 data: 5 5 4 4 5 5 6
24/30 Write block 275 len 16384 data: 12 10 11 11
25/30 Read block 499 len 40960
26/30 Flush
27/30 Write block 178 len 32768 data: 10 13 12 15 13 11 8 6
28/30 Read block 415 len 8192
29/30 Read block 119 len 36864
30/30 Read block 30 len 8192
----------------------------------------------------------------
Crucible gen:78 GIO:true work queues: Upstairs:0 downstairs:4
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1026 Acked Write 8 Done Done Done false
1027 Acked Read 2 Done Done Done false
1028 Acked Read 9 Done Done Done false
1029 Acked Read 2 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 4 4 4 12
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1025 1025 1025
Downstairs last five completed: 1025 1024 1023 1022 1021
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:33.892639314Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1494}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 75 75 75 136 136 136 F F F
3 060-079 73 73 73 133 133 133 F F F
4 080-099 78 78 78 140 140 140 F F F
5 100-119 76 76 76 137 137 137 F F F
6 120-139 70 70 70 128 128 128 F F F
7 140-159 75 75 75 136 136 136 F F F
8 160-179 78 78 78 142 142 142 T T T
9 180-199 78 78 78 142 142 142 T T T
10 200-219 78 78 78 140 140 140 F F F
11 220-239 78 78 78 141 141 141 F F F
12 240-259 78 78 78 142 142 142 F F F
13 260-279 78 78 78 142 142 142 F F F
14 280-299 71 71 71 130 130 130 F F F
15 300-319 78 78 78 142 142 142 F F F
16 320-339 76 76 76 137 137 137 F F F
17 340-359 77 77 77 139 139 139 F F F
18 360-379 78 78 78 141 141 141 F F F
19 380-399 78 78 78 141 141 141 F F F
20 400-419 75 75 75 136 136 136 F F F
21 420-439 77 77 77 139 139 139 F F F
22 440-459 75 75 75 136 136 136 F F F
23 460-479 72 72 72 132 132 132 F F F
24 480-499 78 78 78 141 141 141 F F F
25 500-519 78 78 78 142 142 142 F F F
26 520-539 77 77 77 139 139 139 F F F
27 540-559 78 78 78 142 142 142 F F F
28 560-579 76 76 76 138 138 138 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 78, Max flush: 142
A Difference in extent metadata is expected here
On loop 39
Verifying data now
No change after verify
Loop: 39 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 75 75 75 136 136 136 F F F
3 060-079 73 73 73 133 133 133 F F F
4 080-099 78 78 78 140 140 140 F F F
5 100-119 76 76 76 137 137 137 F F F
6 120-139 70 70 70 128 128 128 F F F
7 140-159 75 75 75 136 136 136 F F F
8 160-179 79 79 79 143 143 143 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 78 78 78 140 140 140 F F F
11 220-239 78 78 78 141 141 141 F F F
12 240-259 78 78 78 142 142 142 F F F
13 260-279 78 78 78 142 142 142 F F F
14 280-299 71 71 71 130 130 130 F F F
15 300-319 78 78 78 142 142 142 F F F
16 320-339 76 76 76 137 137 137 F F F
17 340-359 77 77 77 139 139 139 F F F
18 360-379 78 78 78 141 141 141 F F F
19 380-399 78 78 78 141 141 141 F F F
20 400-419 75 75 75 136 136 136 F F F
21 420-439 77 77 77 139 139 139 F F F
22 440-459 75 75 75 136 136 136 F F F
23 460-479 72 72 72 132 132 132 F F F
24 480-499 78 78 78 141 141 141 F F F
25 500-519 78 78 78 142 142 142 F F F
26 520-539 77 77 77 139 139 139 F F F
27 540-559 78 78 78 142 142 142 F F F
28 560-579 76 76 76 138 138 138 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 79, Max flush: 143
Begin loop 40
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.136684497Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.18260538Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.183778474Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.183831032Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.183847291Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.183862552Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502}
{"msg":"Crucible 8149edd6-8442-4678-9d74-6bcf5e40206a has session id: a26a7d1c-72f9-4f91-85a5-181ec8458b16","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.184530148Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"Upstairs opts: Upstairs UUID: 8149edd6-8442-4678-9d74-6bcf5e40206a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.184597926Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"Crucible stats registered with UUID: 8149edd6-8442-4678-9d74-6bcf5e40206a","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.184625549Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:38.184645697Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185232274Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"task":"crutest"}
{"msg":"The guest has requested activation with gen:80","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185267802Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502}
{"msg":"8149edd6-8442-4678-9d74-6bcf5e40206a active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185329564Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185355088Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"0","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185382321Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"0","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185397211Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"1","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185456575Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"1","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185506384Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"2","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185548548Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"2","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185585505Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"io task","client":"2","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185617935Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"io task","client":"0","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.18588428Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"io task","client":"1","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"ds_connection connected from Ok(127.0.0.1:41717)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185931891Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"io task","client":"2","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"ds_connection connected from Ok(127.0.0.1:48745)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185959483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"io task","client":"0","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"ds_connection connected from Ok(127.0.0.1:43692)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.185985178Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"io task","client":"1","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.186467389Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"0","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.186510013Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"2","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.186574802Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"1","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.187119465Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"0","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.187164357Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"1","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.187288411Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"2","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.187746916Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.187877695Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"[0]R flush_numbers[0..12]: [133, 118, 136, 133, 140, 137, 128, 136, 143, 143, 140, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.187952409Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"downstairs","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"[0]R generation[0..12]: [73, 66, 75, 73, 78, 76, 70, 75, 79, 79, 78, 78]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.187972397Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"downstairs","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.187986898Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"downstairs","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"[1]R flush_numbers[0..12]: [133, 118, 136, 133, 140, 137, 128, 136, 143, 143, 140, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188000619Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"downstairs","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"[1]R generation[0..12]: [73, 66, 75, 73, 78, 76, 70, 75, 79, 79, 78, 78]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188013791Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"downstairs","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188027343Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"downstairs","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"[2]R flush_numbers[0..12]: [133, 118, 136, 133, 140, 137, 128, 136, 143, 143, 140, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188040754Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"downstairs","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"[2]R generation[0..12]: [73, 66, 75, 73, 78, 76, 70, 75, 79, 79, 78, 78]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188054426Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"downstairs","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188068148Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"downstairs","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"Max found gen is 80","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.18808072Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"downstairs","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"Generation requested: 80 >= found:80","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188093702Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"downstairs","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"Next flush: 144","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188106474Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"downstairs","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188122384Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"":"downstairs","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188135406Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188146959Z","hostnameWait for a query_work_queue command to finish before sending IO
":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"0","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188172873Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"1","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188185955Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"client":"2","":"downstairs","session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188198717Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188210449Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"8149edd6-8442-4678-9d74-6bcf5e40206a is now active with session: a26a7d1c-72f9-4f91-85a5-181ec8458b16","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188222822Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188234225Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502,"session_id":"a26a7d1c-72f9-4f91-85a5-181ec8458b16"}
{"msg":"The guest has finished waiting for activation with:80","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:38.188245867Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 289 len 32768 data: 5 5 6 7 7 6 7 8
02/30 Read block 426 len 16384
03/30 Read block 11 len 32768
04/30 Write block 112 len 32768 data: 7 6 6 8 11 12 10 11
05/30 Write block 396 len 28672 data: 6 5 4 3 5 5 6
06/30 Read block 113 len 28672
07/30 Write block 543 len 12288 data: 5 5 4
08/30 Flush
09/30 Write block 105 len 32768 data: 11 10 7 7 7 6 7 8
10/30 Read block 409 len 20480
11/30 Write block 375 len 16384 data: 12 12 11 11
12/30 Flush
13/30 Write block 253 len 28672 data: 4 6 7 7 8 5 4
14/30 Write block 472 len 36864 data: 6 6 6 6 5 6 7 5 6
15/30 Flush
16/30 Flush
17/30 Write block 68 len 32768 data: 9 7 9 9 7 8 8 7
18/30 Write block 160 len 36864 data: 3 8 7 6 9 9 10 9 8
19/30 Read block 456 len 8192
20/30 Write block 67 len 28672 data: 9 10 8 10 10 8 9
21/30 Read block 326 len 20480
22/30 Write block 385 len 4096 data: 4
23/30 Flush
24/30 Write block 274 len 40960 data: 12 13 11 12 12 9 8 8 7 4
25/30 Write block 398 len 12288 data: 5 4 6
26/30 Read block 107 len 4096
27/30 Read block 473 len 40960
28/30 Write block 78 len 32768 data: 5 4 7 8 9 9 10 12
29/30 Write block 141 len 40960 data: 6 7 8 8 10 11 9 9 10 9
30/30 Write block 170 len 40960 data: 7 7 7 6 6 10 10 10 11 14
----------------------------------------------------------------
Crucible gen:80 GIO:true work queues: Upstairs:0 downstairs:7
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1023 Acked Write 10 Done Done Done false
1024 Acked Write 3 Done Done Done false
1025 Acked Read 1 Done Done Done false
1026 Acked Read 10 Done Done Done false
1027 Acked Write 8 Done Done Done false
1028 Acked Write 10 Sent Sent Sent false
1029 Acked Write 10 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 5 5 5 15
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1022 1022 1022
Downstairs last five completed: 1022 1021 1020 1019 1018
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:38.209342174Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1502}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 75 75 75 136 136 136 F F F
3 060-079 80 80 80 148 148 148 T T T
4 080-099 78 78 78 140 140 140 T T T
5 100-119 80 80 80 145 145 145 F F F
6 120-139 70 70 70 128 128 128 F F F
7 140-159 75 75 75 136 136 136 T T T
8 160-179 80 80 80 148 148 148 T T T
9 180-199 79 79 79 143 143 143 F F F
10 200-219 78 78 78 140 140 140 F F F
11 220-239 78 78 78 141 141 141 F F F
12 240-259 80 80 80 146 146 146 F F F
13 260-279 78 78 78 142 142 142 T T T
14 280-299 80 80 80 144 144 144 T T T
15 300-319 78 78 78 142 142 142 F F F
16 320-339 76 76 76 137 137 137 F F F
17 340-359 77 77 77 139 139 139 F F F
18 360-379 80 80 80 145 145 145 F F F
19 380-399 80 80 80 148 148 148 T T T
20 400-419 80 80 80 144 144 144 T T T
21 420-439 77 77 77 139 139 139 F F F
22 440-459 75 75 75 136 136 136 F F F
23 460-479 80 80 80 146 146 146 F F F
24 480-499 80 80 80 146 146 146 F F F
25 500-519 78 78 78 142 142 142 F F F
26 520-539 77 77 77 139 139 139 F F F
27 540-559 80 80 80 144 144 144 F F F
28 560-579 76 76 76 138 138 138 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 80, Max flush: 148
A Difference in extent metadata is expected here
On loop 40
Verifying data now
No change after verify
Loop: 40 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 75 75 75 136 136 136 F F F
3 060-079 81 81 81 149 149 149 F F F
4 080-099 81 81 81 149 149 149 F F F
5 100-119 80 80 80 145 145 145 F F F
6 120-139 70 70 70 128 128 128 F F F
7 140-159 81 81 81 149 149 149 F F F
8 160-179 81 81 81 149 149 149 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 78 78 78 140 140 140 F F F
11 220-239 78 78 78 141 141 141 F F F
12 240-259 80 80 80 146 146 146 F F F
13 260-279 81 81 81 149 149 149 F F F
14 280-299 81 81 81 149 149 149 F F F
15 300-319 78 78 78 142 142 142 F F F
16 320-339 76 76 76 137 137 137 F F F
17 340-359 77 77 77 139 139 139 F F F
18 360-379 80 80 80 145 145 145 F F F
19 380-399 81 81 81 149 149 149 F F F
20 400-419 81 81 81 149 149 149 F F F
21 420-439 77 77 77 139 139 139 F F F
22 440-459 75 75 75 136 136 136 F F F
23 460-479 80 80 80 146 146 146 F F F
24 480-499 80 80 80 146 146 146 F F F
25 500-519 78 78 78 142 142 142 F F F
26 520-539 77 77 77 139 139 139 F F F
27 540-559 80 80 80 144 144 144 F F F
28 560-579 76 76 76 138 138 138 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 81, Max flush: 149
Begin loop 41
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.474888002Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:42.519545828Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.520250352Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.553289143Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.554173915Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.554213141Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.55422997Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.554254865Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510}
{"msg":"Crucible c66e2388-f712-498d-8302-2a5de6996e76 has session id: 56b15086-c56f-48f5-b47a-1decaaa6927b","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.554837094Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"Upstairs opts: Upstairs UUID: c66e2388-f712-498d-8302-2a5de6996e76, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.554900235Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"Crucible stats registered with UUID: c66e2388-f712-498d-8302-2a5de6996e76","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.55492479Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:42.554944718Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555390322Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"task":"crutest"}
{"msg":"The guest has requested activation with gen:82","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555416086Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510}
{"msg":"c66e2388-f712-498d-8302-2a5de6996e76 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555486272Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555503921Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"0","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555527137Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"0","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555540609Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"1","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.55555442Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"1","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555567382Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"2","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555582123Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"2","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555594845Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"io task","client":"2","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555607987Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"io task","client":"0","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555621039Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"io task","client":"1","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"ds_connection connected from Ok(127.0.0.1:62766)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555810162Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"io task","client":"0","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"ds_connection connected from Ok(127.0.0.1:37889)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555950545Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"io task","client":"1","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"ds_connection connected from Ok(127.0.0.1:36913)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.555998155Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"io task","client":"2","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.556570841Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"2","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.556610866Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"1","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.556769778Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"0","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.557268749Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"1","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.557310822Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"2","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.557554412Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"0","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.557862991Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.55796059Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"[0]R flush_numbers[0..12]: [133, 118, 136, 149, 149, 145, 128, 149, 149, 143, 140, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558091329Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"downstairs","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"[0]R generation[0..12]: [73, 66, 75, 81, 81, 80, 70, 81, 81, 79, 78, 78]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558111517Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"downstairs","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558126568Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"downstairs","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"[1]R flush_numbers[0..12]: [133, 118, 136, 149, 149, 145, 128, 149, 149, 143, 140, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558140459Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"downstairs","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"[1]R generation[0..12]: [73, 66, 75, 81, 81, 80, 70, 81, 81, 79, 78, 78]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.55815545Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"downstairs","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558172959Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"downstairs","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"[2]R flush_numbers[0..12]: [133, 118, 136, 149, 149, 145, 128, 149, 149, 143, 140, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.55818669Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"downstairs","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"[2]R generation[0..12]: [73, 66, 75, 81, 81, 80, 70, 81, 81, 79, 78, 78]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558199702Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"downstairs","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558213883Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"downstairs","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"Max found gen is 82","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558226296Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"Wait for a query_work_queue command to finish before sending IO
downstairs","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"Generation requested: 82 >= found:82","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558245544Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"downstairs","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"Next flush: 150","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558258046Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"downstairs","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558270438Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"":"downstairs","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558283091Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558294683Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"0","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558307485Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"1","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558319808Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"client":"2","":"downstairs","session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.55833236Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558344103Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"c66e2388-f712-498d-8302-2a5de6996e76 is now active with session: 56b15086-c56f-48f5-b47a-1decaaa6927b","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558356405Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.558369917Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510,"session_id":"56b15086-c56f-48f5-b47a-1decaaa6927b"}
{"msg":"The guest has finished waiting for activation with:82","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:42.55838175Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 213 len 12288 data: 6 5 7
02/30 Read block 168 len 32768
03/30 Read block 45 len 4096
04/30 Read block 260 len 24576
05/30 Read block 559 len 28672
06/30 Read block 415 len 16384
07/30 Read block 280 len 16384
08/30 Read block 472 len 8192
09/30 Write block 337 len 12288 data: 4 5 7
10/30 Write block 542 len 20480 data: 4 6 6 5 6
11/30 Read block 252 len 24576
12/30 Read block 281 len 40960
13/30 Read block 265 len 8192
14/30 Read block 259 len 16384
15/30 Write block 108 len 24576 data: 8 8 7 8 9 7
16/30 Write block 386 len 40960 data: 5 5 6 7 8 8 9 9 7 7
17/30 Read block 209 len 36864
18/30 Write block 98 len 32768 data: 7 7 9 10 9 9 12 12
19/30 Read block 444 len 16384
20/30 Write block 348 len 20480 data: 7 8 7 6 6
21/30 Write block 532 len 4096 data: 11
22/30 Read block 133 len 24576
23/30 Read block 390 len 12288
24/30 Read block 316 len 32768
25/30 Write block 158 len 8192 data: 6 5
26/30 Read block 69 len 8192
27/30 Read block 227 len 32768
28/30 Write block 141 len 20480 data: 7 8 9 9 11
29/30 Read block 492 len 40960
30/30 Read block 503 len 12288
----------------------------------------------------------------
Crucible gen:82 GIO:true work queues: Upstairs:0 downstairs:30
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Write 3 Done Done Done false
1001 Acked Read 8 Done Done Done false
1002 Acked Read 1 Done Done Done false
1003 Acked Read 6 Done Done Done false
1004 Acked Read 7 Done Done Done false
1005 Acked Read 4 Done Done Done false
1006 Acked Read 4 Done Done Done false
1007 Acked Read 2 Done Done Done false
1008 Acked Write 3 Done Done Done false
1009 Acked Write 5 Done Done Done false
1010 Acked Read 6 Done Done Done false
1011 Acked Read 10 Done Done Done false
1012 Acked Read 2 Done Done Done false
1013 Acked Read 4 Done Done Done false
1014 Acked Write 6 Done Done Done false
1015 Acked Write 10 Done Done Done false
1016 Acked Read 9 Done Done Done false
1017 Acked Write 8 Done Done Done false
1018 Acked Read 4 Done Done Done false
1019 Acked Write 5 Done Done Done false
1020 Acked Write 1 Done Done Done false
1021 Acked Read 6 Done Done Done false
1022 Acked Read 3 Done Done Done false
1023 Acked Read 8 Done Done Done false
1024 Acked Write 2 Done Done Done false
1025 Acked Read 2 Done Done Done false
1026 Acked Read 8 Done Done Done false
1027 Acked Write 5 Done Done Done false
1028 Acked Read 10 Done Done Done false
1029 Acked Read 3 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 1 2
Done 30 29 29 88
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 0 0 0
Downstairs last five completed:
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:42.58061778Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1510}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 75 75 75 136 136 136 F F F
3 060-079 81 81 81 149 149 149 F F F
4 080-099 81 81 81 149 149 149 T T T
5 100-119 80 80 80 145 145 145 T T T
6 120-139 70 70 70 128 128 128 F F F
7 140-159 81 81 81 149 149 149 T T T
8 160-179 81 81 81 149 149 149 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 78 78 78 140 140 140 T T T
11 220-239 78 78 78 141 141 141 F F F
12 240-259 80 80 80 146 146 146 F F F
13 260-279 81 81 81 149 149 149 F F F
14 280-299 81 81 81 149 149 149 F F F
15 300-319 78 78 78 142 142 142 F F F
16 320-339 76 76 76 137 137 137 T T T
17 340-359 77 77 77 139 139 139 T T T
18 360-379 80 80 80 145 145 145 F F F
19 380-399 81 81 81 149 149 149 T T T
20 400-419 81 81 81 149 149 149 F F F
21 420-439 77 77 77 139 139 139 F F F
22 440-459 75 75 75 136 136 136 F F F
23 460-479 80 80 80 146 146 146 F F F
24 480-499 80 80 80 146 146 146 F F F
25 500-519 78 78 78 142 142 142 F F F
26 520-539 77 77 77 139 139 139 T T T
27 540-559 80 80 80 144 144 144 T T T
28 560-579 76 76 76 138 138 138 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 81, Max flush: 149
A Difference in extent metadata is expected here
On loop 41
Verifying data now
No change after verify
Loop: 41 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 75 75 75 136 136 136 F F F
3 060-079 81 81 81 149 149 149 F F F
4 080-099 82 82 82 150 150 150 F F F
5 100-119 82 82 82 150 150 150 F F F
6 120-139 70 70 70 128 128 128 F F F
7 140-159 82 82 82 150 150 150 F F F
8 160-179 81 81 81 149 149 149 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 82 82 82 150 150 150 F F F
11 220-239 78 78 78 141 141 141 F F F
12 240-259 80 80 80 146 146 146 F F F
13 260-279 81 81 81 149 149 149 F F F
14 280-299 81 81 81 149 149 149 F F F
15 300-319 78 78 78 142 142 142 F F F
16 320-339 82 82 82 150 150 150 F F F
17 340-359 82 82 82 150 150 150 F F F
18 360-379 80 80 80 145 145 145 F F F
19 380-399 82 82 82 150 150 150 F F F
20 400-419 81 81 81 149 149 149 F F F
21 420-439 77 77 77 139 139 139 F F F
22 440-459 75 75 75 136 136 136 F F F
23 460-479 80 80 80 146 146 146 F F F
24 480-499 80 80 80 146 146 146 F F F
25 500-519 78 78 78 142 142 142 F F F
26 520-539 82 82 82 150 150 150 F F F
27 540-559 82 82 82 150 150 150 F F F
28 560-579 76 76 76 138 138 138 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 82, Max flush: 150
Begin loop 42
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.891619551Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.935735962Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.936754391Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.936797934Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.936812685Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.936826666Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518}
{"msg":"Crucible 2e3c1fa4-3ade-4c7b-923b-a2171a2aa9c1 has session id: 96c8dfa9-e500-4588-9cbc-e64bd477f8c8","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.937564989Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"Upstairs opts: Upstairs UUID: 2e3c1fa4-3ade-4c7b-923b-a2171a2aa9c1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.937614438Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"Crucible stats registered with UUID: 2e3c1fa4-3ade-4c7b-923b-a2171a2aa9c1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.937657902Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:46.937679798Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938162349Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"task":"crutest"}
{"msg":"The guest has requested activation with gen:84","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938198877Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518}
{"msg":"2e3c1fa4-3ade-4c7b-923b-a2171a2aa9c1 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938278557Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938297416Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"0","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938324749Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"0","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938355799Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"1","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938388639Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"1","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.93841928Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"2","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938467121Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"2","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938498721Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"io task","client":"2","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938528842Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"io task","client":"0","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938554876Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"io task","client":"1","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"ds_connection connected from Ok(127.0.0.1:42617)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938587276Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"io task","client":"0","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"ds_connection connected from Ok(127.0.0.1:59249)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938777289Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"io task","client":"1","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49410)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.938901062Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"io task","client":"2","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.939596481Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"1","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.939655785Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"0","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.940380146Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"1","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.940423689Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"0","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.940592374Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"2","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941143313Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941180161Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"2","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941203826Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"[0]R flush_numbers[0..12]: [133, 118, 136, 149, 150, 150, 128, 150, 149, 143, 150, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941736266Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"downstairs","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"[0]R generation[0..12]: [73, 66, 75, 81, 82, 82, 70, 82, 81, 79, 82, 78]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941764349Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"downstairs","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941780199Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"downstairs","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"[1]R flush_numbers[0..12]: [133, 118, 136, 149, 150, 150, 128, 150, 149, 143, 150, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.94179419Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"downstairs","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"[1]R generation[0..12]: [73, 66, 75, 81, 82, 82, 70, 82, 81, 79, 82, 78]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941807662Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"downstairs","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941821233Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"downstairs","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"[2]R flush_numbers[0..12]: [133, 118, 136, 149, 150, 150, 128, 150, 149, 143, 150, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941834585Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"downstairs","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"[2]R generation[0..12]: [73, 66, 75, 81, 82, 82, 70, 82, 81, 79, 82, 78]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941847447Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"downstairs","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941860789Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"downstairs","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"Max found gen is 83","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941873181Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"downstairs","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"Generation requested: 84 >= found:83","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941889681Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"downstairs","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"Next flush: 151","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941902623Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"downstairs","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941916005Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"":"downstairs","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"Wait for a query_work_queue command to finish before sending IO
}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941942109Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941954711Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"0","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.941973639Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"1","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.94198755Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"client":"2","":"downstairs","session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.942000253Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.942012135Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"2e3c1fa4-3ade-4c7b-923b-a2171a2aa9c1 is now active with session: 96c8dfa9-e500-4588-9cbc-e64bd477f8c8","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.942024558Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.94203622Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518,"session_id":"96c8dfa9-e500-4588-9cbc-e64bd477f8c8"}
{"msg":"The guest has finished waiting for activation with:84","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:46.942049152Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 126 len 32768 data: 4 4 3 3 3 5 4 5
02/30 Flush
03/30 Write block 147 len 20480 data: 10 10 11 10 8
04/30 Read block 47 len 32768
05/30 Flush
06/30 Write block 478 len 28672 data: 8 6 7 5 5 6 6
07/30 Read block 76 len 24576
08/30 Write block 438 len 20480 data: 5 4 5 4 4
09/30 Read block 452 len 20480
10/30 Write block 566 len 12288 data: 5 5 5
11/30 Flush
12/30 Write block 287 len 32768 data: 4 5 6 6 7 8 8 7
13/30 Read block 31 len 12288
14/30 Read block 59 len 36864
15/30 Read block 441 len 40960
16/30 Write block 440 len 32768 data: 6 5 5 5 5 6 6 6
17/30 Read block 548 len 12288
18/30 Write block 174 len 8192 data: 7 11
19/30 Read block 194 len 8192
20/30 Read block 110 len 36864
21/30 Read block 335 len 24576
22/30 Write block 540 len 4096 data: 6
23/30 Flush
24/30 Write block 246 len 40960 data: 8 10 10 8 5 5 4 5 7 8
25/30 Read block 268 len 36864
26/30 Read block 448 len 40960
27/30 Write block 400 len 32768 data: 7 6 7 7 7 7 6 5
28/30 Write block 481 len 32768 data: 6 6 7 7 6 7 10 9
29/30 Read block 571 len 24576
30/30 Read block 31 len 12288
----------------------------------------------------------------
Crucible gen:84 GIO:true work queues: Upstairs:0 downstairs:7
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1023 Acked Write 10 Done Done Done false
1024 Acked Read 9 Done Done Done false
1025 Acked Read 10 Done Done Done false
1026 Acked Write 8 Done Done Done false
1027 Acked Write 8 Done Done Done false
1028 Acked Read 6 Done Done Done false
1029 Acked Read 3 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 1 2
Done 7 6 6 19
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1022 1022 1022
Downstairs last five completed: 1022 1021 1020 1019 1018
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:46.964734635Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1518}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 75 75 75 136 136 136 F F F
3 060-079 81 81 81 149 149 149 F F F
4 080-099 82 82 82 150 150 150 F F F
5 100-119 82 82 82 150 150 150 F F F
6 120-139 84 84 84 151 151 151 F F F
7 140-159 84 84 84 152 152 152 F F F
8 160-179 84 84 84 154 154 154 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 82 82 82 150 150 150 F F F
11 220-239 78 78 78 141 141 141 F F F
12 240-259 80 80 80 146 146 146 T T T
13 260-279 81 81 81 149 149 149 F F F
14 280-299 84 84 84 154 154 154 F F F
15 300-319 78 78 78 142 142 142 F F F
16 320-339 82 82 82 150 150 150 F F F
17 340-359 82 82 82 150 150 150 F F F
18 360-379 80 80 80 145 145 145 F F F
19 380-399 82 82 82 150 150 150 F F F
20 400-419 81 81 81 149 149 149 T T T
21 420-439 84 84 84 153 153 153 F F F
22 440-459 84 84 84 154 154 154 F F F
23 460-479 84 84 84 153 153 153 F F F
24 480-499 84 84 84 153 153 153 T T T
25 500-519 78 78 78 142 142 142 F F F
26 520-539 82 82 82 150 150 150 F F F
27 540-559 84 84 84 154 154 154 F F F
28 560-579 84 84 84 153 153 153 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 84, Max flush: 154
A Difference in extent metadata is expected here
On loop 42
Verifying data now
No change after verify
Loop: 42 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 75 75 75 136 136 136 F F F
3 060-079 81 81 81 149 149 149 F F F
4 080-099 82 82 82 150 150 150 F F F
5 100-119 82 82 82 150 150 150 F F F
6 120-139 84 84 84 151 151 151 F F F
7 140-159 84 84 84 152 152 152 F F F
8 160-179 84 84 84 154 154 154 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 82 82 82 150 150 150 F F F
11 220-239 78 78 78 141 141 141 F F F
12 240-259 85 85 85 155 155 155 F F F
13 260-279 81 81 81 149 149 149 F F F
14 280-299 84 84 84 154 154 154 F F F
15 300-319 78 78 78 142 142 142 F F F
16 320-339 82 82 82 150 150 150 F F F
17 340-359 82 82 82 150 150 150 F F F
18 360-379 80 80 80 145 145 145 F F F
19 380-399 82 82 82 150 150 150 F F F
20 400-419 85 85 85 155 155 155 F F F
21 420-439 84 84 84 153 153 153 F F F
22 440-459 84 84 84 154 154 154 F F F
23 460-479 84 84 84 153 153 153 F F F
24 480-499 85 85 85 155 155 155 F F F
25 500-519 78 78 78 142 142 142 F F F
26 520-539 82 82 82 150 150 150 F F F
27 540-559 84 84 84 154 154 154 F F F
28 560-579 84 84 84 153 153 153 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 85, Max flush: 155
Begin loop 43
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.214716762Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.258539465Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.259217715Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.259259629Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.259275039Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.25929001Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526}
{"msg":"Crucible 2044e91d-905e-45de-ad0f-63264f238f13 has session id: 9f370a09-b8f4-4412-a58c-b69349ad05ff","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.259860387Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"Upstairs opts: Upstairs UUID: 2044e91d-905e-45de-ad0f-63264f238f13, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.259904819Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"Crucible stats registered with UUID: 2044e91d-905e-45de-ad0f-63264f238f13","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.259932732Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:51.259955428Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.260316504Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"task":"crutest"}
{"msg":"The guest has requested activation with gen:86","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.260341529Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526}
{"msg":"2044e91d-905e-45de-ad0f-63264f238f13 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.260412844Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.260430633Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"0","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.260454139Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"0","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.2604691Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"1","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.260494544Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"1","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.260520258Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"2","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.260545552Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"2","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.260571676Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"io task","client":"2","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.2605974Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"io task","client":"0","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.260621106Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"io task","client":"1","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49554)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.26066199Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"io task","client":"0","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:51.260722603Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"io task","client":"1","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:51.260760829Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"io task","client":"1","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"ds_connection connected from Ok(127.0.0.1:59660)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.260895126Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"io task","client":"2","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:39:51.260934921Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.260958527Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"io task","client":"1","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.261427586Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"2","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.26145413Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"0","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.261950512Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"0","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.262080062Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"2","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.26253584Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:39:51.262666189Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.262145272Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"io task","client":"1","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49608)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.262217228Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"io task","client":"1","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.263428038Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"1","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264132262Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"1","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"[0]R flush_numbers[0..12]: [133, 118, 136, 149, 150, 150, 151, 152, 154, 143, 150, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264783169Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"downstairs","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"[0]R generation[0..12]: [73, 66, 75, 81, 82, 82, 84, 84, 84, 79, 82, 78]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264811552Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"downstairs","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264827851Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"downstairs","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"[1]R flush_numbers[0..12]: [133, 118, 136, 149, 150, 150, 151, 152, 154, 143, 150, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264841903Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"downstairs","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"[1]R generation[0..12]: [73, 66, 75, 81, 82, 82, 84, 84, 84, 79, 82, 78]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264855684Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"downstairs","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264869685Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"downstairs","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"[2]R flush_numbers[0..12]: [133, 118, 136, 149, 150, 150, 151, 152, 154, 143, 150, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264883847Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"downstairs","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"[2]R generation[0..12]: [73, 66, 75, 81, 82, 82, 84, 84, 84, 79, 82, 78]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264897358Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"downstairs","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.26491139Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"downstairs","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"Max found gen is 86","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264924911Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"downstairs","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"Generation requested: 86 >= found:86","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264938273Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"downstairs","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"Next flush: 156","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264951045Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"downstairs","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264964187Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"":"downstairs","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264977029Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.264989371Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"0","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.265002543Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"1","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.265020432Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"client":"2","":"downstairs","session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,Wait for a query_work_queue command to finish before sending IO
"time":"2025-01-27T19:40:01.265034343Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.26505602Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"2044e91d-905e-45de-ad0f-63264f238f13 is now active with session: 9f370a09-b8f4-4412-a58c-b69349ad05ff","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.265068862Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.265081065Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526,"session_id":"9f370a09-b8f4-4412-a58c-b69349ad05ff"}
{"msg":"The guest has finished waiting for activation with:86","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:01.265093097Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 48 len 16384 data: 3 4 5 6
02/30 Write block 327 len 24576 data: 10 7 6 5 4 5
03/30 Read block 492 len 24576
04/30 Write block 261 len 4096 data: 7
05/30 Write block 161 len 24576 data: 9 8 7 10 10 11
06/30 Write block 372 len 40960 data: 8 9 11 13 13 12 12 11 7 5
07/30 Write block 476 len 40960 data: 6 7 9 7 8 7 7 8 8 7
08/30 Read block 405 len 24576
09/30 Flush
10/30 Flush
11/30 Flush
12/30 Flush
13/30 Write block 327 len 40960 data: 11 8 7 6 5 6 4 3 4 4
14/30 Write block 510 len 28672 data: 6 7 6 4 4 5 5
15/30 Write block 44 len 8192 data: 5 5
16/30 Write block 56 len 20480 data: 7 7 7 6 6
17/30 Write block 85 len 12288 data: 13 11 8
18/30 Write block 471 len 36864 data: 6 7 7 7 7 7 8 10 8
19/30 Read block 565 len 8192
20/30 Read block 272 len 16384
21/30 Flush
22/30 Write block 152 len 24576 data: 8 13 12 7 8 6
23/30 Read block 61 len 20480
24/30 Write block 139 len 36864 data: 7 7 8 9 10 10 12 12 11
25/30 Write block 539 len 4096 data: 6
26/30 Write block 223 len 24576 data: 5 4 2 3 3 4
27/30 Read block 248 len 40960
28/30 Read block 401 len 20480
29/30 Write block 105 len 8192 data: 13 11
30/30 Write block 390 len 12288 data: 9 9 10
----------------------------------------------------------------
Crucible gen:86 GIO:true work queues: Upstairs:0 downstairs:9
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1021 Acked Write 6 Done Done Done false
1022 Acked Read 5 Done Done Done false
1023 Acked Write 9 Done Done Done false
1024 Acked Write 1 Done Done Done false
1025 Acked Write 6 Done Done Done false
1026 Acked Read 10 Done Done Done false
1027 Acked Read 5 Done Done Done false
1028 Acked Write 2 Sent Sent Sent false
1029 Acked Write 3 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 7 7 7 21
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1020 1020 1020
Downstairs last five completed: 1020 1019 1018 1017 1016
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:01.284182696Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1526}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 86 86 86 160 160 160 F F F
3 060-079 86 86 86 160 160 160 F F F
4 080-099 86 86 86 160 160 160 F F F
5 100-119 82 82 82 150 150 150 T T T
6 120-139 84 84 84 151 151 151 T T T
7 140-159 84 84 84 152 152 152 T T T
8 160-179 86 86 86 156 156 156 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 82 82 82 150 150 150 F F F
11 220-239 78 78 78 141 141 141 T T T
12 240-259 85 85 85 155 155 155 F F F
13 260-279 86 86 86 156 156 156 F F F
14 280-299 84 84 84 154 154 154 F F F
15 300-319 78 78 78 142 142 142 F F F
16 320-339 86 86 86 160 160 160 F F F
17 340-359 82 82 82 150 150 150 F F F
18 360-379 86 86 86 156 156 156 F F F
19 380-399 86 86 86 156 156 156 T T T
20 400-419 85 85 85 155 155 155 F F F
21 420-439 84 84 84 153 153 153 F F F
22 440-459 84 84 84 154 154 154 F F F
23 460-479 86 86 86 160 160 160 F F F
24 480-499 86 86 86 156 156 156 F F F
25 500-519 86 86 86 160 160 160 F F F
26 520-539 82 82 82 150 150 150 T T T
27 540-559 84 84 84 154 154 154 F F F
28 560-579 84 84 84 153 153 153 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 86, Max flush: 160
A Difference in extent metadata is expected here
On loop 43
Verifying data now
No change after verify
Loop: 43 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 86 86 86 160 160 160 F F F
3 060-079 86 86 86 160 160 160 F F F
4 080-099 86 86 86 160 160 160 F F F
5 100-119 87 87 87 161 161 161 F F F
6 120-139 87 87 87 161 161 161 F F F
7 140-159 87 87 87 161 161 161 F F F
8 160-179 86 86 86 156 156 156 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 82 82 82 150 150 150 F F F
11 220-239 87 87 87 161 161 161 F F F
12 240-259 85 85 85 155 155 155 F F F
13 260-279 86 86 86 156 156 156 F F F
14 280-299 84 84 84 154 154 154 F F F
15 300-319 78 78 78 142 142 142 F F F
16 320-339 86 86 86 160 160 160 F F F
17 340-359 82 82 82 150 150 150 F F F
18 360-379 86 86 86 156 156 156 F F F
19 380-399 87 87 87 161 161 161 F F F
20 400-419 85 85 85 155 155 155 F F F
21 420-439 84 84 84 153 153 153 F F F
22 440-459 84 84 84 154 154 154 F F F
23 460-479 86 86 86 160 160 160 F F F
24 480-499 86 86 86 156 156 156 F F F
25 500-519 86 86 86 160 160 160 F F F
26 520-539 87 87 87 161 161 161 F F F
27 540-559 84 84 84 154 154 154 F F F
28 560-579 84 84 84 153 153 153 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 87, Max flush: 161
Begin loop 44
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.54045096Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:05.584866606Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.585662323Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.618830014Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.620018708Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.62006466Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.620081949Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.620099938Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534}
{"msg":"Crucible b6bf9cf7-3431-46f9-98c5-4a5956a84d08 has session id: 8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.620813046Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"Upstairs opts: Upstairs UUID: b6bf9cf7-3431-46f9-98c5-4a5956a84d08, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.620851333Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"Crucible stats registered with UUID: b6bf9cf7-3431-46f9-98c5-4a5956a84d08","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.620866893Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:05.620879825Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.621309669Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"task":"crutest"}
{"msg":"The guest has requested activation with gen:88","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.621334284Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534}
{"msg":"b6bf9cf7-3431-46f9-98c5-4a5956a84d08 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.621415313Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.621432613Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"0","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.621457737Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"0","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.621471349Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"1","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.6214853Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"1","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.621498472Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"2","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.621512393Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"2","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.621525025Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"io task","client":"2","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.621538417Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"io task","client":"0","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.621607734Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"io task","client":"1","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"ds_connection connected from Ok(127.0.0.1:35223)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.621707432Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"io task","client":"0","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"ds_connection connected from Ok(127.0.0.1:56321)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.621748257Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"io task","client":"2","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"ds_connection connected from Ok(127.0.0.1:44497)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.62177668Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"io task","client":"1","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.622646451Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"2","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.622727321Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"0","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.622751406Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"1","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.623307332Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"2","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.623467753Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"1","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.623497894Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"0","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.623920802Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.62400477Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"[0]R flush_numbers[0..12]: [133, 118, 160, 160, 160, 161, 161, 161, 156, 143, 150, 161]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624145423Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"downstairs","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"[0]R generation[0..12]: [73, 66, 86, 86, 86, 87, 87, 87, 86, 79, 82, 87]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624164461Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"downstairs","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624178792Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"downstairs","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"[1]R flush_numbers[0..12]: [133, 118, 160, 160, 160, 161, 161, 161, 156, 143, 150, 161]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624192594Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"downstairs","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"[1]R generation[0..12]: [73, 66, 86, 86, 86, 87, 87, 87, 86, 79, 82, 87]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624206235Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"downstairs","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624219887Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"downstairs","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"[2]R flush_numbers[0..12]: [133, 118, 160, 160, 160, 161, 161, 161, 156, 143, 150, 161]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624233318Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"downstairs","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"[2]R generation[0..12]: [73, 66, 86, 86, 86, 87, 87, 87, 86, 79, 82, 87]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.62424672Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"downstairs","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.62426372Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"downstairs","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"Max found gen is 88","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624276542Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"downstairs","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"Generation requested: 88 >= found:88","v":0,"name":"crucible",Wait for a query_work_queue command to finish before sending IO
"level":30,"time":"2025-01-27T19:40:05.62429564Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"downstairs","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"Next flush: 162","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624308502Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"downstairs","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624321194Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"":"downstairs","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624333926Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624345469Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"0","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624358271Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"1","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624370813Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"client":"2","":"downstairs","session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624383575Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624395348Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"b6bf9cf7-3431-46f9-98c5-4a5956a84d08 is now active with session: 8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.62440779Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624422591Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534,"session_id":"8fbe2c5d-7e8e-41b3-a6ac-ebbfc96662fa"}
{"msg":"The guest has finished waiting for activation with:88","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:05.624434624Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 223 len 36864 data: 6 5 3 4 4 5 3 4 4
02/30 Read block 315 len 36864
03/30 Write block 278 len 40960 data: 13 10 9 9 8 5 4 4 4 5
04/30 Write block 291 len 36864 data: 8 9 9 8 8 9 9 7 8
05/30 Read block 201 len 8192
06/30 Read block 27 len 16384
07/30 Write block 432 len 8192 data: 10 10
08/30 Write block 416 len 32768 data: 3 5 6 7 7 5 5 7
09/30 Read block 362 len 28672
10/30 Read block 538 len 40960
11/30 Write block 240 len 16384 data: 3 3 4 5
12/30 Flush
13/30 Flush
14/30 Write block 165 len 12288 data: 11 12 10
15/30 Write block 566 len 32768 data: 6 6 6 5 7 6 7 7
16/30 Read block 413 len 12288
17/30 Write block 230 len 8192 data: 5 5
18/30 Write block 7 len 16384 data: 9 7 8 8
19/30 Read block 252 len 4096
20/30 Read block 97 len 24576
21/30 Read block 192 len 8192
22/30 Read block 343 len 32768
23/30 Read block 114 len 8192
24/30 Write block 496 len 4096 data: 9
25/30 Write block 416 len 20480 data: 4 6 7 8 8
26/30 Write block 252 len 8192 data: 5 6
27/30 Read block 413 len 24576
28/30 Write block 302 len 20480 data: 8 10 9 10 11
29/30 Read block 109 len 20480
30/30 Write block 551 len 32768 data: 10 10 7 9 8 8 7 6
----------------------------------------------------------------
Crucible gen:88 GIO:true work queues: Upstairs:0 downstairs:17
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1013 Acked Write 3 Done Done Done false
1014 Acked Write 8 Done Done Done false
1015 Acked Read 3 Done Done Done false
1016 Acked Write 2 Done Done Done false
1017 Acked Write 4 Done Done Done false
1018 Acked Read 1 Done Done Done false
1019 Acked Read 6 Done Done Done false
1020 Acked Read 2 Done Done Done false
1021 Acked Read 8 Done Done Done false
1022 Acked Read 2 Done Done Done false
1023 Acked Write 1 Done Done Done false
1024 Acked Write 5 Done Done Done false
1025 Acked Write 2 Done Done Done false
1026 Acked Read 6 Done Done Done false
1027 Acked Write 5 Done Done Done false
1028 Acked Read 5 Done Done Done false
1029 Acked Write 8 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 16 16 16 48
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1012 1012 1012
Downstairs last five completed: 1012 1011 1010 1009 1008
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:05.644609191Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1534}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 73 73 73 133 133 133 T T T
1 020-039 66 66 66 118 118 118 F F F
2 040-059 86 86 86 160 160 160 F F F
3 060-079 86 86 86 160 160 160 F F F
4 080-099 86 86 86 160 160 160 F F F
5 100-119 87 87 87 161 161 161 F F F
6 120-139 87 87 87 161 161 161 F F F
7 140-159 87 87 87 161 161 161 F F F
8 160-179 86 86 86 156 156 156 T T T
9 180-199 79 79 79 143 143 143 F F F
10 200-219 82 82 82 150 150 150 F F F
11 220-239 88 88 88 162 162 162 T T T
12 240-259 88 88 88 162 162 162 T T T
13 260-279 88 88 88 162 162 162 F F F
14 280-299 88 88 88 162 162 162 F F F
15 300-319 78 78 78 142 142 142 T T T
16 320-339 86 86 86 160 160 160 F F F
17 340-359 82 82 82 150 150 150 F F F
18 360-379 86 86 86 156 156 156 F F F
19 380-399 87 87 87 161 161 161 F F F
20 400-419 88 88 88 162 162 162 T T T
21 420-439 88 88 88 162 162 162 T T T
22 440-459 84 84 84 154 154 154 F F F
23 460-479 86 86 86 160 160 160 F F F
24 480-499 86 86 86 156 156 156 T T T
25 500-519 86 86 86 160 160 160 F F F
26 520-539 87 87 87 161 161 161 F F F
27 540-559 84 84 84 154 154 154 T T T
28 560-579 84 84 84 153 153 153 T T T
29 580-599 75 75 75 136 136 136 F F F
Max gen: 88, Max flush: 162
A Difference in extent metadata is expected here
On loop 44
Verifying data now
No change after verify
Loop: 44 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 89 89 89 163 163 163 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 86 86 86 160 160 160 F F F
3 060-079 86 86 86 160 160 160 F F F
4 080-099 86 86 86 160 160 160 F F F
5 100-119 87 87 87 161 161 161 F F F
6 120-139 87 87 87 161 161 161 F F F
7 140-159 87 87 87 161 161 161 F F F
8 160-179 89 89 89 163 163 163 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 82 82 82 150 150 150 F F F
11 220-239 89 89 89 163 163 163 F F F
12 240-259 89 89 89 163 163 163 F F F
13 260-279 88 88 88 162 162 162 F F F
14 280-299 88 88 88 162 162 162 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 86 86 86 160 160 160 F F F
17 340-359 82 82 82 150 150 150 F F F
18 360-379 86 86 86 156 156 156 F F F
19 380-399 87 87 87 161 161 161 F F F
20 400-419 89 89 89 163 163 163 F F F
21 420-439 89 89 89 163 163 163 F F F
22 440-459 84 84 84 154 154 154 F F F
23 460-479 86 86 86 160 160 160 F F F
24 480-499 89 89 89 163 163 163 F F F
25 500-519 86 86 86 160 160 160 F F F
26 520-539 87 87 87 161 161 161 F F F
27 540-559 89 89 89 163 163 163 F F F
28 560-579 89 89 89 163 163 163 F F F
29 580-599 75 75 75 136 136 136 F F F
Max gen: 89, Max flush: 163
Begin loop 45
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:09.958319234Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:10.004217931Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.004932269Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.03751789Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.038458837Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.038496084Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.038514613Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.038530693Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542}
{"msg":"Crucible b30edaf9-283e-4f87-9626-267256f28ba5 has session id: 29158d16-b769-41ac-9dd3-c26ec278763d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039089207Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"Upstairs opts: Upstairs UUID: b30edaf9-283e-4f87-9626-267256f28ba5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039134829Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"Crucible stats registered with UUID: b30edaf9-283e-4f87-9626-267256f28ba5","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039160863Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:10.039184158Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039702138Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"task":"crutest"}
{"msg":"The guest has requested activation with gen:90","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039725513Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542}
{"msg":"b30edaf9-283e-4f87-9626-267256f28ba5 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039798438Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039815647Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"0","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039838513Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"0","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039851785Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"1","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039865556Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"1","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039878258Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"2","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039896077Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"2","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039909179Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"io task","client":"0","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.039994246Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"io task","client":"1","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.040024468Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"io task","client":"2","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:59287)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.04010162Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"io task","client":"1","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:51014)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.040242183Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"io task","client":"0","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:64224)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.040270216Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"io task","client":"2","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.040873392Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"2","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.040912228Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"1","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.041052222Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"0","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.04145881Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"1","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.041577536Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"2","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.041750649Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"0","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042160265Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042187388Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"[0]R flush_numbers[0..12]: [163, 118, 160, 160, 160, 161, 161, 161, 163, 143, 150, 163]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042339294Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"downstairs","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"[0]R generation[0..12]: [89, 66, 86, 86, 86, 87, 87, 87, 89, 79, 82, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042358662Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"downstairs","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042373024Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"downstairs","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"[1]R flush_numbers[0..12]: [163, 118, 160, 160, 160, 161, 161, 161, 163, 143, 150, 163]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042386735Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"downstairs","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"[1]R generation[0..12]: [89, 66, 86, 86, 86, 87, 87, 87, 89, 79, 82, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042401006Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"downstairs","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042414828Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"downstairs","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"[2]R flush_numbers[0..12]: [163, 118, 160, 160, 160, 161, 161, 161, 163, 143, 150, 163]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.04242809Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"downstairs","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"[2]R generation[0..12]: [89, 66, 86, 86, 86, 87, 87, 87, 89, 79, 82, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042441581Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"downstairs","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042455123Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"downstairs","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"Max found gen is 90","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042467695Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"downstairs","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"Wait for a query_work_queue command to finish before sending IO
Generation requested: 90 >= found:90","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042487642Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"downstairs","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"Next flush: 164","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042500245Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"downstairs","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042512857Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"":"downstairs","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042525539Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.04253992Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"0","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042553142Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"1","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042566194Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"client":"2","":"downstairs","session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042579376Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042591228Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"b30edaf9-283e-4f87-9626-267256f28ba5 is now active with session: 29158d16-b769-41ac-9dd3-c26ec278763d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.04260445Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042616213Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542,"session_id":"29158d16-b769-41ac-9dd3-c26ec278763d"}
{"msg":"The guest has finished waiting for activation with:90","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:10.042628075Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 480 len 36864 data: 9 8 8 9 9 8 8 11 10
02/30 Read block 425 len 36864
03/30 Write block 73 len 28672 data: 10 9 8 5 5 6 5
04/30 Write block 337 len 32768 data: 5 6 8 7 8 8 7 8
05/30 Read block 305 len 8192
06/30 Read block 305 len 20480
07/30 Read block 429 len 16384
08/30 Read block 350 len 36864
09/30 Flush
10/30 Write block 86 len 32768 data: 12 9 8 7 9 8 11 11
11/30 Read block 116 len 12288
12/30 Read block 72 len 28672
13/30 Read block 520 len 12288
14/30 Read block 470 len 12288
15/30 Read block 410 len 4096
16/30 Write block 110 len 16384 data: 8 9 10 8
17/30 Write block 573 len 40960 data: 8 7 6 6 5 4 4 5 5 5
18/30 Read block 375 len 12288
19/30 Write block 499 len 8192 data: 7 5
20/30 Read block 306 len 20480
21/30 Read block 554 len 8192
22/30 Read block 175 len 12288
23/30 Read block 234 len 24576
24/30 Read block 256 len 28672
25/30 Read block 291 len 36864
26/30 Read block 380 len 36864
27/30 Read block 390 len 4096
28/30 Read block 457 len 24576
29/30 Write block 460 len 12288 data: 7 6 7
30/30 Write block 1 len 28672 data: 3 4 7 7 8 8 10
----------------------------------------------------------------
Crucible gen:90 GIO:true work queues: Upstairs:0 downstairs:21
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1009 Acked Write 8 Done Done Done false
1010 Acked Read 3 Done Done Done false
1011 Acked Read 7 Done Done Done false
1012 Acked Read 3 Done Done Done false
1013 Acked Read 3 Done Done Done false
1014 Acked Read 1 Done Done Done false
1015 Acked Write 4 Done Done Done false
1016 Acked Write 10 Done Done Done false
1017 Acked Read 3 Done Done Done false
1018 Acked Write 2 Done Done Done false
1019 Acked Read 5 Done Done Done false
1020 Acked Read 2 Done Done Done false
1021 Acked Read 3 Done Done Done false
1022 Acked Read 6 Done Done Done false
1023 Acked Read 7 Done Done Done false
1024 Acked Read 9 Done Done Done false
1025 Acked Read 9 Done Done Done false
1026 Acked Read 1 Done Done Done false
1027 Acked Read 6 Done Done Done false
1028 Acked Write 3 Done Done Done false
1029 Acked Write 7 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 20 20 20 60
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1008 1008 1008
Downstairs last five completed: 1008 1007 1006 1005 1004
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:10.064456498Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1542}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 89 89 89 163 163 163 T T T
1 020-039 66 66 66 118 118 118 F F F
2 040-059 86 86 86 160 160 160 F F F
3 060-079 90 90 90 164 164 164 F F F
4 080-099 86 86 86 160 160 160 T T T
5 100-119 87 87 87 161 161 161 T T T
6 120-139 87 87 87 161 161 161 F F F
7 140-159 87 87 87 161 161 161 F F F
8 160-179 89 89 89 163 163 163 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 82 82 82 150 150 150 F F F
11 220-239 89 89 89 163 163 163 F F F
12 240-259 89 89 89 163 163 163 F F F
13 260-279 88 88 88 162 162 162 F F F
14 280-299 88 88 88 162 162 162 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 90 90 90 164 164 164 F F F
17 340-359 90 90 90 164 164 164 F F F
18 360-379 86 86 86 156 156 156 F F F
19 380-399 87 87 87 161 161 161 F F F
20 400-419 89 89 89 163 163 163 F F F
21 420-439 89 89 89 163 163 163 F F F
22 440-459 84 84 84 154 154 154 F F F
23 460-479 86 86 86 160 160 160 T T T
24 480-499 90 90 90 164 164 164 T T T
25 500-519 86 86 86 160 160 160 T T T
26 520-539 87 87 87 161 161 161 F F F
27 540-559 89 89 89 163 163 163 F F F
28 560-579 89 89 89 163 163 163 T T T
29 580-599 75 75 75 136 136 136 T T T
Max gen: 90, Max flush: 164
A Difference in extent metadata is expected here
On loop 45
Verifying data now
No change after verify
Loop: 45 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 91 91 91 165 165 165 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 86 86 86 160 160 160 F F F
3 060-079 90 90 90 164 164 164 F F F
4 080-099 91 91 91 165 165 165 F F F
5 100-119 91 91 91 165 165 165 F F F
6 120-139 87 87 87 161 161 161 F F F
7 140-159 87 87 87 161 161 161 F F F
8 160-179 89 89 89 163 163 163 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 82 82 82 150 150 150 F F F
11 220-239 89 89 89 163 163 163 F F F
12 240-259 89 89 89 163 163 163 F F F
13 260-279 88 88 88 162 162 162 F F F
14 280-299 88 88 88 162 162 162 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 90 90 90 164 164 164 F F F
17 340-359 90 90 90 164 164 164 F F F
18 360-379 86 86 86 156 156 156 F F F
19 380-399 87 87 87 161 161 161 F F F
20 400-419 89 89 89 163 163 163 F F F
21 420-439 89 89 89 163 163 163 F F F
22 440-459 84 84 84 154 154 154 F F F
23 460-479 91 91 91 165 165 165 F F F
24 480-499 91 91 91 165 165 165 F F F
25 500-519 91 91 91 165 165 165 F F F
26 520-539 87 87 87 161 161 161 F F F
27 540-559 89 89 89 163 163 163 F F F
28 560-579 91 91 91 165 165 165 F F F
29 580-599 91 91 91 165 165 165 F F F
Max gen: 91, Max flush: 165
Begin loop 46
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.369665736Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.416057978Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.417614484Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.417666362Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.417684201Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.41770144Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550}
{"msg":"Crucible 2d58a59e-aedb-4f79-ae12-4655b6a1f716 has session id: b963b2ee-1674-4c35-8268-f49a3fd3eb6d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.418425012Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"Upstairs opts: Upstairs UUID: 2d58a59e-aedb-4f79-ae12-4655b6a1f716, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.418478039Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"Crucible stats registered with UUID: 2d58a59e-aedb-4f79-ae12-4655b6a1f716","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.418504913Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:14.418520033Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.418990422Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"task":"crutest"}
{"msg":"The guest has requested activation with gen:92","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.419089021Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550}
{"msg":"2d58a59e-aedb-4f79-ae12-4655b6a1f716 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.419114765Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.419129116Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"0","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.419163535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"0","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.419190798Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"1","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.4192199Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"1","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.419350259Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"2","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.419385777Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"2","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.419414719Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"io task","client":"2","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.41944567Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"io task","client":"0","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.419475052Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"io task","client":"1","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47952)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.419506772Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"io task","client":"2","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53256)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.419534665Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"io task","client":"1","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:41077)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.41955814Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"io task","client":"0","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.420235691Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"0","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.420429711Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"1","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.42089843Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"0","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.421085315Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"1","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.421278465Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"2","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.421612618Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.42173928Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422025133Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"2","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"[0]R flush_numbers[0..12]: [165, 118, 160, 164, 165, 165, 161, 161, 163, 143, 150, 163]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422575972Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"downstairs","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"[0]R generation[0..12]: [91, 66, 86, 90, 91, 91, 87, 87, 89, 79, 82, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422599237Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"downstairs","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422614818Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"downstairs","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"[1]R flush_numbers[0..12]: [165, 118, 160, 164, 165, 165, 161, 161, 163, 143, 150, 163]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422628919Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"downstairs","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"[1]R generation[0..12]: [91, 66, 86, 90, 91, 91, 87, 87, 89, 79, 82, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422642371Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"downstairs","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422655972Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"downstairs","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"[2]R flush_numbers[0..12]: [165, 118, 160, 164, 165, 165, 161, 161, 163, 143, 150, 163]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422674101Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"downstairs","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"[2]R generation[0..12]: [91, 66, 86, 90, 91, 91, 87, 87, 89, 79, 82, 89]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422687933Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"downstairs","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","levelWait for a query_work_queue command to finish before sending IO
":30,"time":"2025-01-27T19:40:14.4227088Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"downstairs","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"Max found gen is 92","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422721712Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"downstairs","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"Generation requested: 92 >= found:92","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422734374Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"downstairs","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"Next flush: 166","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422746966Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"downstairs","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422759398Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"":"downstairs","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.42277252Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422784323Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"0","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422797295Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"1","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422809997Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"client":"2","":"downstairs","session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422822739Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422834522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"2d58a59e-aedb-4f79-ae12-4655b6a1f716 is now active with session: b963b2ee-1674-4c35-8268-f49a3fd3eb6d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422847114Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422858807Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550,"session_id":"b963b2ee-1674-4c35-8268-f49a3fd3eb6d"}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
{"msg":"The guest has finished waiting for activation with:92","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:14.422875776Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 556 len 16384 data: 9 8 7 8
02/30 Read block 49 len 20480
03/30 Read block 73 len 16384
04/30 Read block 559 len 8192
05/30 Read block 32 len 24576
06/30 Read block 464 len 20480
07/30 Write block 490 len 40960 data: 8 9 11 9 6 10 10 8 8 8
08/30 Write block 202 len 32768 data: 7 7 9 9 8 8 9 8
09/30 Write block 558 len 8192 data: 8 9
10/30 Write block 590 len 16384 data: 4 3 3 4
11/30 Write block 229 len 40960 data: 4 6 6 3 3 4 4 3 4 2
12/30 Write block 563 len 8192 data: 5 5
13/30 Flush
14/30 Write block 455 len 4096 data: 10
15/30 Read block 30 len 24576
16/30 Flush
17/30 Write block 46 len 16384 data: 4 4 4 5
18/30 Write block 392 len 24576 data: 11 10 8 8 7 6
19/30 Flush
20/30 Write block 577 len 4096 data: 6
21/30 Read block 493 len 12288
22/30 Read block 591 len 36864
23/30 Read block 430 len 20480
24/30 Read block 353 len 4096
25/30 Read block 0 len 16384
26/30 Write block 211 len 8192 data: 6 6
27/30 Write block 429 len 36864 data: 7 9 8 11 11 11 12 9 5
28/30 Read block 309 len 4096
29/30 Read block 465 len 40960
30/30 Write block 525 len 36864 data: 8 7 10 12 11 12 12 12 11
----------------------------------------------------------------
Crucible gen:92 GIO:true work queues: Upstairs:0 downstairs:11
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1019 Acked Write 1 Done Done Done false
1020 Acked Read 3 Done Done Done false
1021 Acked Read 9 Done Done Done false
1022 Acked Read 5 Done Done Done false
1023 Acked Read 1 Done Done Done false
1024 Acked Read 4 Done Done Done false
1025 Acked Write 2 Done Done Done false
1026 Acked Write 9 Done Done Done false
1027 Acked Read 1 Done Done Done false
1028 Acked Read 10 Done Done Done false
1029 Acked Write 9 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 10 10 10 30
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1018 1018 1018
Downstairs last five completed: 1018 1017 1016 1015 1014
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:14.443613005Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1550}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 91 91 91 165 165 165 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 92 92 92 168 168 168 F F F
3 060-079 90 90 90 164 164 164 F F F
4 080-099 91 91 91 165 165 165 F F F
5 100-119 91 91 91 165 165 165 F F F
6 120-139 87 87 87 161 161 161 F F F
7 140-159 87 87 87 161 161 161 F F F
8 160-179 89 89 89 163 163 163 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 92 92 92 166 166 166 T T T
11 220-239 92 92 92 166 166 166 F F F
12 240-259 89 89 89 163 163 163 F F F
13 260-279 88 88 88 162 162 162 F F F
14 280-299 88 88 88 162 162 162 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 90 90 90 164 164 164 F F F
17 340-359 90 90 90 164 164 164 F F F
18 360-379 86 86 86 156 156 156 F F F
19 380-399 92 92 92 168 168 168 F F F
20 400-419 89 89 89 163 163 163 F F F
21 420-439 89 89 89 163 163 163 T T T
22 440-459 92 92 92 167 167 167 F F F
23 460-479 91 91 91 165 165 165 F F F
24 480-499 92 92 92 166 166 166 F F F
25 500-519 91 91 91 165 165 165 F F F
26 520-539 87 87 87 161 161 161 T T T
27 540-559 92 92 92 166 166 166 F F F
28 560-579 92 92 92 166 166 166 T T T
29 580-599 92 92 92 166 166 166 F F F
Max gen: 92, Max flush: 168
A Difference in extent metadata is expected here
On loop 46
Verifying data now
No change after verify
Loop: 46 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 91 91 91 165 165 165 F F F
1 020-039 66 66 66 118 118 118 F F F
2 040-059 92 92 92 168 168 168 F F F
3 060-079 90 90 90 164 164 164 F F F
4 080-099 91 91 91 165 165 165 F F F
5 100-119 91 91 91 165 165 165 F F F
6 120-139 87 87 87 161 161 161 F F F
7 140-159 87 87 87 161 161 161 F F F
8 160-179 89 89 89 163 163 163 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 93 93 93 169 169 169 F F F
11 220-239 92 92 92 166 166 166 F F F
12 240-259 89 89 89 163 163 163 F F F
13 260-279 88 88 88 162 162 162 F F F
14 280-299 88 88 88 162 162 162 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 90 90 90 164 164 164 F F F
17 340-359 90 90 90 164 164 164 F F F
18 360-379 86 86 86 156 156 156 F F F
19 380-399 92 92 92 168 168 168 F F F
20 400-419 89 89 89 163 163 163 F F F
21 420-439 93 93 93 169 169 169 F F F
22 440-459 92 92 92 167 167 167 F F F
23 460-479 91 91 91 165 165 165 F F F
24 480-499 92 92 92 166 166 166 F F F
25 500-519 91 91 91 165 165 165 F F F
26 520-539 93 93 93 169 169 169 F F F
27 540-559 92 92 92 166 166 166 F F F
28 560-579 93 93 93 169 169 169 F F F
29 580-599 92 92 92 166 166 166 F F F
Max gen: 93, Max flush: 169
Begin loop 47
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.694045623Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.739618612Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.740570982Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.740611017Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.740625878Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.740640089Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558}
{"msg":"Crucible cf78411b-9424-4bf3-a944-7ac6dbaeb7c3 has session id: 9613c328-0149-43e5-8fce-4a922217d2ee","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.741154161Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"Upstairs opts: Upstairs UUID: cf78411b-9424-4bf3-a944-7ac6dbaeb7c3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.741197104Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"Crucible stats registered with UUID: cf78411b-9424-4bf3-a944-7ac6dbaeb7c3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.741221259Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:18.741241267Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.741782462Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"task":"crutest"}
{"msg":"The guest has requested activation with gen:94","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.741807896Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558}
{"msg":"cf78411b-9424-4bf3-a944-7ac6dbaeb7c3 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.741862262Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.741887907Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"0","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.741921226Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"0","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.741945801Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"1","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.741968856Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"1","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.74199475Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"2","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.742025891Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"2","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.742052745Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"io task","client":"2","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.742081807Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"io task","client":"0","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.742105492Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"io task","client":"1","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"ds_connection connected from Ok(127.0.0.1:62418)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.742128587Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"io task","client":"2","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"ds_connection connected from Ok(127.0.0.1:51666)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.742154142Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"io task","client":"0","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"ds_connection connected from Ok(127.0.0.1:60888)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.742351729Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"io task","client":"1","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.742874665Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"1","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.74291498Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"0","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.743071703Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"2","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.74349768Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"1","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.74362582Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"0","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.743663217Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"2","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744107452Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744197246Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"[0]R flush_numbers[0..12]: [165, 118, 168, 164, 165, 165, 161, 161, 163, 143, 169, 166]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744313055Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"downstairs","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"[0]R generation[0..12]: [91, 66, 92, 90, 91, 91, 87, 87, 89, 79, 93, 92]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.74433647Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"downstairs","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744351591Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"downstairs","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"[1]R flush_numbers[0..12]: [165, 118, 168, 164, 165, 165, 161, 161, 163, 143, 169, 166]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744365762Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"downstairs","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"[1]R generation[0..12]: [91, 66, 92, 90, 91, 91, 87, 87, 89, 79, 93, 92]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744379234Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"downstairs","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744392945Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"downstairs","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"[2]R flush_numbers[0..12]: [165, 118, 168, 164, 165, 165, 161, 161, 163, 143, 169, 166]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744406547Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"downstairs","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"[2]R generation[0..12]: [91, 66, 92, 90, 91, 91, 87, 87, 89, 79, 93, 92]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744419739Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"downstairs","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.74443333Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"downstairs","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"Max found gen is 94","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744446012Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"downstairs","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"Generation requested: 94 >= found:94","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744459084Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"downstairs","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"Next flush: 170","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:40:18.744471696Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"downstairs","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744490485Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"":"downstairs","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744503517Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.74451526Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"0","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744528332Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"1","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744542783Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"client":"2","":"downstairs","session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744555874Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744567957Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"cf78411b-9424-4bf3-a944-7ac6dbaeb7c3 is now active with session: 9613c328-0149-43e5-8fce-4a922217d2ee","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744580569Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744592322Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558,"session_id":"9613c328-0149-43e5-8fce-4a922217d2ee"}
{"msg":"The guest has finished waiting for activation with:94","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:18.744604105Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 543 len 36864 data: 7 7 6 7 7 7 9 8 11
02/30 Read block 116 len 4096
03/30 Write block 484 len 8192 data: 10 9
04/30 Flush
05/30 Write block 65 len 12288 data: 10 10 10
06/30 Read block 190 len 12288
07/30 Write block 119 len 24576 data: 12 9 9 8 5 6
08/30 Read block 543 len 32768
09/30 Flush
10/30 Write block 556 len 8192 data: 10 9
11/30 Read block 455 len 40960
12/30 Write block 129 len 16384 data: 4 4 6 5
13/30 Read block 283 len 8192
14/30 Read block 483 len 28672
15/30 Read block 586 len 32768
16/30 Read block 170 len 24576
17/30 Read block 318 len 24576
18/30 Write block 16 len 4096 data: 2
19/30 Read block 487 len 4096
20/30 Write block 212 len 36864 data: 7 7 6 8 6 6 7 6 6
21/30 Read block 167 len 4096
22/30 Write block 157 len 20480 data: 7 7 6 4 10
23/30 Read block 114 len 8192
24/30 Flush
25/30 Write block 344 len 8192 data: 9 7
26/30 Write block 22 len 12288 data: 8 10 11
27/30 Write block 419 len 32768 data: 9 9 6 6 8 7 7 6
28/30 Read block 88 len 16384
29/30 Write block 494 len 8192 data: 7 11
30/30 Write block 524 len 4096 data: 9
----------------------------------------------------------------
Crucible gen:94 GIO:true work queues: Upstairs:0 downstairs:6
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1024 Acked Write 2 Done Done Done false
1025 Acked Write 3 Done Done Done false
1026 Acked Write 8 Done Done Done false
1027 Acked Read 4 Done Done Done false
1028 Acked Write 2 Sent Sent Sent false
1029 Acked Write 1 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 4 4 4 12
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1023 1023 1023
Downstairs last five completed: 1023 1022 1021 1020 1019
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:18.764472831Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1558}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 94 94 94 172 172 172 F F F
1 020-039 66 66 66 118 118 118 T T T
2 040-059 92 92 92 168 168 168 F F F
3 060-079 94 94 94 171 171 171 F F F
4 080-099 91 91 91 165 165 165 F F F
5 100-119 94 94 94 171 171 171 F F F
6 120-139 94 94 94 172 172 172 F F F
7 140-159 94 94 94 172 172 172 F F F
8 160-179 94 94 94 172 172 172 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 94 94 94 172 172 172 F F F
11 220-239 94 94 94 172 172 172 F F F
12 240-259 89 89 89 163 163 163 F F F
13 260-279 88 88 88 162 162 162 F F F
14 280-299 88 88 88 162 162 162 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 90 90 90 164 164 164 F F F
17 340-359 90 90 90 164 164 164 T T T
18 360-379 86 86 86 156 156 156 F F F
19 380-399 92 92 92 168 168 168 F F F
20 400-419 89 89 89 163 163 163 T T T
21 420-439 93 93 93 169 169 169 T T T
22 440-459 92 92 92 167 167 167 F F F
23 460-479 91 91 91 165 165 165 F F F
24 480-499 94 94 94 170 170 170 T T T
25 500-519 91 91 91 165 165 165 F F F
26 520-539 93 93 93 169 169 169 T T T
27 540-559 94 94 94 172 172 172 F F F
28 560-579 93 93 93 169 169 169 F F F
29 580-599 92 92 92 166 166 166 F F F
Max gen: 94, Max flush: 172
A Difference in extent metadata is expected here
On loop 47
Verifying data now
No change after verify
Loop: 47 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 94 94 94 172 172 172 F F F
1 020-039 95 95 95 173 173 173 F F F
2 040-059 92 92 92 168 168 168 F F F
3 060-079 94 94 94 171 171 171 F F F
4 080-099 91 91 91 165 165 165 F F F
5 100-119 94 94 94 171 171 171 F F F
6 120-139 94 94 94 172 172 172 F F F
7 140-159 94 94 94 172 172 172 F F F
8 160-179 94 94 94 172 172 172 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 94 94 94 172 172 172 F F F
11 220-239 94 94 94 172 172 172 F F F
12 240-259 89 89 89 163 163 163 F F F
13 260-279 88 88 88 162 162 162 F F F
14 280-299 88 88 88 162 162 162 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 90 90 90 164 164 164 F F F
17 340-359 95 95 95 173 173 173 F F F
18 360-379 86 86 86 156 156 156 F F F
19 380-399 92 92 92 168 168 168 F F F
20 400-419 95 95 95 173 173 173 F F F
21 420-439 95 95 95 173 173 173 F F F
22 440-459 92 92 92 167 167 167 F F F
23 460-479 91 91 91 165 165 165 F F F
24 480-499 95 95 95 173 173 173 F F F
25 500-519 91 91 91 165 165 165 F F F
26 520-539 95 95 95 173 173 173 F F F
27 540-559 94 94 94 172 172 172 F F F
28 560-579 93 93 93 169 169 169 F F F
29 580-599 92 92 92 166 166 166 F F F
Max gen: 95, Max flush: 173
Begin loop 48
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.020673496Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.066086754Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.067108551Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.067152204Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.067168204Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.067184034Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566}
{"msg":"Crucible 311b003d-2de0-44fc-afd0-491f9b384a4d has session id: 16fea44f-4828-47fd-bc22-de5aa6ae25ba","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.06782142Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"Upstairs opts: Upstairs UUID: 311b003d-2de0-44fc-afd0-491f9b384a4d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.067858897Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"Crucible stats registered with UUID: 311b003d-2de0-44fc-afd0-491f9b384a4d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.067876006Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:23.067889928Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.068431132Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"task":"crutest"}
{"msg":"The guest has requested activation with gen:96","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.06846636Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566}
{"msg":"311b003d-2de0-44fc-afd0-491f9b384a4d active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.068521896Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.06854866Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"0","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.068587325Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"0","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.068616467Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"1","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.068651436Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"1","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.068681817Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"2","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.06870978Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"2","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.068732436Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"io task","client":"2","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.068758539Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"io task","client":"0","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.068785893Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"io task","client":"1","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47452)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.068812686Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"io task","client":"2","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"ds_connection connected from Ok(127.0.0.1:63159)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.06883874Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"io task","client":"0","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39013)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.068867862Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"io task","client":"1","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.069531761Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"1","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.069570437Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"0","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.070314136Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"1","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.070352073Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"0","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.071058715Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.071095452Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"2","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.071111213Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.071630111Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"2","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"[0]R flush_numbers[0..12]: [172, 173, 168, 171, 165, 171, 172, 172, 172, 143, 172, 172]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072150579Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"downstairs","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"[0]R generation[0..12]: [94, 95, 92, 94, 91, 94, 94, 94, 94, 79, 94, 94]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072174584Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"downstairs","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072189825Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"downstairs","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"[1]R flush_numbers[0..12]: [172, 173, 168, 171, 165, 171, 172, 172, 172, 143, 172, 172]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072203636Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"downstairs","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"[1]R generation[0..12]: [94, 95, 92, 94, 91, 94, 94, 94, 94, 79, 94, 94]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072217188Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"downstairs","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072230729Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"downstairs","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"[2]R flush_numbers[0..12]: [172, 173, 168, 171, 165, 171, 172, 172, 172, 143, 172, 172]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072244021Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"downstairs","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"[2]R generation[0..12]: [94, 95, 92, 94, 91, 94, 94, 94, 94, 79, 94, 94]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072257053Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"downstairs","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072270785Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"downstairs","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"Max found gen is 96","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072283347Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"downstairs","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"Generation requested: 96 >= found:96","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072296059Z","hostname":"Wait for a query_work_queue command to finish before sending IO
w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"downstairs","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"Next flush: 174","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072317635Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"downstairs","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072330288Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"":"downstairs","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.07234326Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072354983Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"0","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072367784Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"1","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072380487Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"client":"2","":"downstairs","session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072393109Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072405132Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"311b003d-2de0-44fc-afd0-491f9b384a4d is now active with session: 16fea44f-4828-47fd-bc22-de5aa6ae25ba","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072418123Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072429956Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566,"session_id":"16fea44f-4828-47fd-bc22-de5aa6ae25ba"}
{"msg":"The guest has finished waiting for activation with:96","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:23.072441829Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 170 len 28672 data: 8 8 8 7 8 12 11
02/30 Read block 591 len 28672
03/30 Write block 462 len 4096 data: 8
04/30 Read block 314 len 20480
05/30 Read block 12 len 28672
06/30 Read block 499 len 8192
07/30 Read block 573 len 24576
08/30 Read block 206 len 40960
09/30 Read block 197 len 28672
10/30 Write block 441 len 8192 data: 6 6
11/30 Read block 297 len 16384
12/30 Read block 209 len 20480
13/30 Read block 371 len 36864
14/30 Flush
15/30 Write block 508 len 32768 data: 7 6 7 8 7 5 5 6
16/30 Read block 516 len 20480
17/30 Read block 528 len 36864
18/30 Read block 516 len 8192
19/30 Read block 193 len 20480
20/30 Flush
21/30 Write block 363 len 40960 data: 5 4 5 5 5 7 9 10 10 9
22/30 Write block 514 len 36864 data: 6 7 6 5 5 6 6 9 9
23/30 Read block 407 len 28672
24/30 Write block 45 len 8192 data: 6 5
25/30 Write block 235 len 8192 data: 5 4
26/30 Flush
27/30 Write block 90 len 24576 data: 10 9 12 12 9 10
28/30 Read block 120 len 28672
29/30 Write block 342 len 20480 data: 9 8 10 8 7
30/30 Read block 7 len 36864
----------------------------------------------------------------
Crucible gen:96 GIO:true work queues: Upstairs:0 downstairs:4
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1026 Acked Write 6 Done Done Done false
1027 Acked Read 7 Done Done Done false
1028 Acked Write 5 Done Done Done false
1029 Acked Read 9 Sent Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 0 1 2
Done 3 4 3 10
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1025 1025 1025
Downstairs last five completed: 1025 1024 1023 1022 1021
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:23.093529051Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1566}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 94 94 94 172 172 172 F F F
1 020-039 95 95 95 173 173 173 F F F
2 040-059 96 96 96 176 176 176 F F F
3 060-079 94 94 94 171 171 171 F F F
4 080-099 91 91 91 165 165 165 T T T
5 100-119 94 94 94 171 171 171 F F F
6 120-139 94 94 94 172 172 172 F F F
7 140-159 94 94 94 172 172 172 F F F
8 160-179 96 96 96 174 174 174 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 94 94 94 172 172 172 F F F
11 220-239 96 96 96 176 176 176 F F F
12 240-259 89 89 89 163 163 163 F F F
13 260-279 88 88 88 162 162 162 F F F
14 280-299 88 88 88 162 162 162 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 90 90 90 164 164 164 F F F
17 340-359 95 95 95 173 173 173 T T T
18 360-379 96 96 96 176 176 176 F F F
19 380-399 92 92 92 168 168 168 F F F
20 400-419 95 95 95 173 173 173 F F F
21 420-439 95 95 95 173 173 173 F F F
22 440-459 96 96 96 174 174 174 F F F
23 460-479 96 96 96 174 174 174 F F F
24 480-499 95 95 95 173 173 173 F F F
25 500-519 96 96 96 176 176 176 F F F
26 520-539 96 96 96 176 176 176 F F F
27 540-559 94 94 94 172 172 172 F F F
28 560-579 93 93 93 169 169 169 F F F
29 580-599 92 92 92 166 166 166 F F F
Max gen: 96, Max flush: 176
A Difference in extent metadata is expected here
On loop 48
Verifying data now
No change after verify
Loop: 48 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 94 94 94 172 172 172 F F F
1 020-039 95 95 95 173 173 173 F F F
2 040-059 96 96 96 176 176 176 F F F
3 060-079 94 94 94 171 171 171 F F F
4 080-099 97 97 97 177 177 177 F F F
5 100-119 94 94 94 171 171 171 F F F
6 120-139 94 94 94 172 172 172 F F F
7 140-159 94 94 94 172 172 172 F F F
8 160-179 96 96 96 174 174 174 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 94 94 94 172 172 172 F F F
11 220-239 96 96 96 176 176 176 F F F
12 240-259 89 89 89 163 163 163 F F F
13 260-279 88 88 88 162 162 162 F F F
14 280-299 88 88 88 162 162 162 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 90 90 90 164 164 164 F F F
17 340-359 97 97 97 177 177 177 F F F
18 360-379 96 96 96 176 176 176 F F F
19 380-399 92 92 92 168 168 168 F F F
20 400-419 95 95 95 173 173 173 F F F
21 420-439 95 95 95 173 173 173 F F F
22 440-459 96 96 96 174 174 174 F F F
23 460-479 96 96 96 174 174 174 F F F
24 480-499 95 95 95 173 173 173 F F F
25 500-519 96 96 96 176 176 176 F F F
26 520-539 96 96 96 176 176 176 F F F
27 540-559 94 94 94 172 172 172 F F F
28 560-579 93 93 93 169 169 169 F F F
29 580-599 92 92 92 166 166 166 F F F
Max gen: 97, Max flush: 177
Begin loop 49
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.334131735Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.380127533Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.381029434Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.381079423Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.381097492Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.381115941Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574}
{"msg":"Crucible b9bfebb4-560a-4ba9-b445-8a41a1fa2212 has session id: b580b3d4-e3a7-4922-b544-b361fe673c09","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.38173009Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"Upstairs opts: Upstairs UUID: b9bfebb4-560a-4ba9-b445-8a41a1fa2212, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.381870433Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"Crucible stats registered with UUID: b9bfebb4-560a-4ba9-b445-8a41a1fa2212","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.3819075Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:27.381928917Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382401115Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"task":"crutest"}
{"msg":"The guest has requested activation with gen:98","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382432126Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574}
{"msg":"b9bfebb4-560a-4ba9-b445-8a41a1fa2212 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382495076Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382512856Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"0","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382536261Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"0","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382549763Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"1","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382563974Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"1","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382581393Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"2","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382606618Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"2","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382632991Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"io task","client":"2","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382658475Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"io task","client":"0","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382684299Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"io task","client":"1","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"ds_connection connected from Ok(127.0.0.1:62072)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382711253Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"io task","client":"2","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49872)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382871693Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"io task","client":"0","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"ds_connection connected from Ok(127.0.0.1:55502)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.382905113Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"io task","client":"1","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.383358422Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"2","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.383563075Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"1","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.383612065Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"0","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384124198Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"2","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384157247Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"0","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384226934Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"1","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384749261Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384774745Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"[0]R flush_numbers[0..12]: [172, 173, 176, 171, 177, 171, 172, 172, 174, 143, 172, 176]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384788336Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"downstairs","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"[0]R generation[0..12]: [94, 95, 96, 94, 97, 94, 94, 94, 96, 79, 94, 96]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384802388Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"downstairs","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384816339Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"downstairs","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"[1]R flush_numbers[0..12]: [172, 173, 176, 171, 177, 171, 172, 172, 174, 143, 172, 176]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384829771Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"downstairs","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"[1]R generation[0..12]: [94, 95, 96, 94, 97, 94, 94, 94, 96, 79, 94, 96]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384842793Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"downstairs","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384856384Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"downstairs","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"[2]R flush_numbers[0..12]: [172, 173, 176, 171, 177, 171, 172, 172, 174, 143, 172, 176]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384869896Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"downstairs","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"[2]R generation[0..12]: [94, 95, 96, 94, 97, 94, 94, 94, 96, 79, 94, 96]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384887245Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"downstairs","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384901506Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"downstairs","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"Max found gen is 98","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384915058Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"downstairs","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"Generation requested: 98 >= found:98","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.38492818Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"downstairs","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"Next flush: 178","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384941042Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"downstairs","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"All extents match","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:40:27.384953454Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"":"downstairs","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384969904Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384981517Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"0","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.384994499Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"1","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.385007491Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"client":"2","":"downstairs","session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.385020313Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.385032065Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"b9bfebb4-560a-4ba9-b445-8a41a1fa2212 is now active with session: b580b3d4-e3a7-4922-b544-b361fe673c09","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.385044468Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.385056131Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574,"session_id":"b580b3d4-e3a7-4922-b544-b361fe673c09"}
{"msg":"The guest has finished waiting for activation with:98","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:27.385068023Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 151 len 16384 data: 9 9 14 13
02/30 Read block 219 len 20480
03/30 Read block 363 len 32768
04/30 Read block 184 len 20480
05/30 Write block 293 len 16384 data: 10 9 9 10
06/30 Flush
07/30 Flush
08/30 Write block 534 len 4096 data: 10
09/30 Read block 241 len 12288
10/30 Write block 218 len 4096 data: 8
11/30 Read block 67 len 4096
12/30 Read block 379 len 36864
13/30 Read block 21 len 8192
14/30 Read block 142 len 12288
15/30 Read block 594 len 4096
16/30 Flush
17/30 Write block 529 len 4096 data: 12
18/30 Read block 130 len 4096
19/30 Read block 167 len 24576
20/30 Write block 175 len 4096 data: 13
21/30 Write block 575 len 4096 data: 7
22/30 Read block 574 len 28672
23/30 Read block 193 len 12288
24/30 Write block 549 len 12288 data: 10 9 12
25/30 Read block 494 len 32768
26/30 Write block 596 len 4096 data: 5
27/30 Write block 248 len 32768 data: 11 9 6 6 6 7 8 9
28/30 Write block 78 len 32768 data: 7 6 8 9 10 10 11 14
29/30 Read block 143 len 40960
30/30 Write block 30 len 8192 data: 10 8
----------------------------------------------------------------
Crucible gen:98 GIO:true work queues: Upstairs:0 downstairs:14
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1016 Acked Write 1 Done Done Done false
1017 Acked Read 1 Done Done Done false
1018 Acked Read 6 Done Done Done false
1019 Acked Write 1 Done Done Done false
1020 Acked Write 1 Done Done Done false
1021 Acked Read 7 Done Done Done false
1022 Acked Read 3 Done Done Done false
1023 Acked Write 3 Done Done Done false
1024 Acked Read 8 Done Done Done false
1025 Acked Write 1 Done Done Done false
1026 Acked Write 8 Done Done Done false
1027 Acked Write 8 Done Done Done false
1028 Acked Read 10 Done Done Sent false
1029 Acked Write 2 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 2 4
Done 13 13 12 38
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1015 1015 1015
Downstairs last five completed: 1015 1014 1013 1012 1011
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:27.403018937Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1574}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 94 94 94 172 172 172 F F F
1 020-039 95 95 95 173 173 173 T T T
2 040-059 96 96 96 176 176 176 F F F
3 060-079 94 94 94 171 171 171 T T T
4 080-099 97 97 97 177 177 177 T T T
5 100-119 94 94 94 171 171 171 F F F
6 120-139 94 94 94 172 172 172 F F F
7 140-159 98 98 98 178 178 178 F F F
8 160-179 96 96 96 174 174 174 T T T
9 180-199 79 79 79 143 143 143 F F F
10 200-219 98 98 98 180 180 180 F F F
11 220-239 96 96 96 176 176 176 F F F
12 240-259 89 89 89 163 163 163 T T T
13 260-279 88 88 88 162 162 162 F F F
14 280-299 98 98 98 178 178 178 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 90 90 90 164 164 164 F F F
17 340-359 97 97 97 177 177 177 F F F
18 360-379 96 96 96 176 176 176 F F F
19 380-399 92 92 92 168 168 168 F F F
20 400-419 95 95 95 173 173 173 F F F
21 420-439 95 95 95 173 173 173 F F F
22 440-459 96 96 96 174 174 174 F F F
23 460-479 96 96 96 174 174 174 F F F
24 480-499 95 95 95 173 173 173 F F F
25 500-519 96 96 96 176 176 176 F F F
26 520-539 98 98 98 180 180 180 T T T
27 540-559 94 94 94 172 172 172 T T T
28 560-579 93 93 93 169 169 169 T T T
29 580-599 92 92 92 166 166 166 T T T
Max gen: 98, Max flush: 180
A Difference in extent metadata is expected here
On loop 49
Verifying data now
No change after verify
Loop: 49 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 94 94 94 172 172 172 F F F
1 020-039 99 99 99 181 181 181 F F F
2 040-059 96 96 96 176 176 176 F F F
3 060-079 99 99 99 181 181 181 F F F
4 080-099 99 99 99 181 181 181 F F F
5 100-119 94 94 94 171 171 171 F F F
6 120-139 94 94 94 172 172 172 F F F
7 140-159 98 98 98 178 178 178 F F F
8 160-179 99 99 99 181 181 181 F F F
9 180-199 79 79 79 143 143 143 F F F
10 200-219 98 98 98 180 180 180 F F F
11 220-239 96 96 96 176 176 176 F F F
12 240-259 99 99 99 181 181 181 F F F
13 260-279 88 88 88 162 162 162 F F F
14 280-299 98 98 98 178 178 178 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 90 90 90 164 164 164 F F F
17 340-359 97 97 97 177 177 177 F F F
18 360-379 96 96 96 176 176 176 F F F
19 380-399 92 92 92 168 168 168 F F F
20 400-419 95 95 95 173 173 173 F F F
21 420-439 95 95 95 173 173 173 F F F
22 440-459 96 96 96 174 174 174 F F F
23 460-479 96 96 96 174 174 174 F F F
24 480-499 95 95 95 173 173 173 F F F
25 500-519 96 96 96 176 176 176 F F F
26 520-539 99 99 99 181 181 181 F F F
27 540-559 99 99 99 181 181 181 F F F
28 560-579 99 99 99 181 181 181 F F F
29 580-599 99 99 99 181 181 181 F F F
Max gen: 99, Max flush: 181
Begin loop 50
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.67471243Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.721140999Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.72212462Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.722198025Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.722225488Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.722249822Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582}
{"msg":"Crucible 34ae69a3-5229-4aa9-8491-6188eaef8ec5 has session id: 331b25d9-76fb-40a2-a869-b507371c3dcb","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.722781863Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"Upstairs opts: Upstairs UUID: 34ae69a3-5229-4aa9-8491-6188eaef8ec5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.722829234Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"Crucible stats registered with UUID: 34ae69a3-5229-4aa9-8491-6188eaef8ec5","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.722857756Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:31.722872607Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723263845Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"task":"crutest"}
{"msg":"The guest has requested activation with gen:100","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723290378Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582}
{"msg":"34ae69a3-5229-4aa9-8491-6188eaef8ec5 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723344595Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723371688Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"0","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723397652Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"0","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723413242Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"1","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723427923Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"1","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723440965Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"2","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723454716Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"2","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723514279Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"io task","client":"2","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723560541Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"io task","client":"0","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723600046Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"io task","client":"1","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"ds_connection connected from Ok(127.0.0.1:51703)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.72362644Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"io task","client":"2","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52644)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723650535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"io task","client":"1","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"ds_connection connected from Ok(127.0.0.1:41432)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.723676219Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"io task","client":"0","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.724426714Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"2","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.724455347Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"0","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.724556124Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"1","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.725189332Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"0","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.725299404Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"2","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.725358647Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"1","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.725922668Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.72604935Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"[0]R flush_numbers[0..12]: [172, 181, 176, 181, 181, 171, 172, 178, 181, 143, 180, 176]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726124173Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"downstairs","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"[0]R generation[0..12]: [94, 99, 96, 99, 99, 94, 94, 98, 99, 79, 98, 96]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726143291Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"downstairs","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726160831Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"downstairs","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"[1]R flush_numbers[0..12]: [172, 181, 176, 181, 181, 171, 172, 178, 181, 143, 180, 176]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726174612Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"downstairs","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"[1]R generation[0..12]: [94, 99, 96, 99, 99, 94, 94, 98, 99, 79, 98, 96]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726188314Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"downstairs","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726202095Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"downstairs","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"[2]R flush_numbers[0..12]: [172, 181, 176, 181, 181, 171, 172, 178, 181, 143, 180, 176]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726215517Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"downstairs","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"[2]R generation[0..12]: [94, 99, 96, 99, 99, 94, 94, 98, 99, 79, 98, 96]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726228878Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"downstairs","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.72624236Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"downstairs","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"Max found gen is 100","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726254922Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"downstairs","":"downstairs","session_id":Wait for a query_work_queue command to finish before sending IO
"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"Generation requested: 100 >= found:100","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726276279Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"downstairs","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"Next flush: 182","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726288881Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"downstairs","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726301344Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"":"downstairs","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726314156Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726325928Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"0","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.72633885Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"1","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726351722Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"client":"2","":"downstairs","session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726366433Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726378396Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"34ae69a3-5229-4aa9-8491-6188eaef8ec5 is now active with session: 331b25d9-76fb-40a2-a869-b507371c3dcb","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726390848Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726402471Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582,"session_id":"331b25d9-76fb-40a2-a869-b507371c3dcb"}
{"msg":"The guest has finished waiting for activation with:100","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:31.726414274Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 275 len 24576 data: 14 12 13 14 11 10
02/30 Read block 72 len 4096
03/30 Write block 593 len 28672 data: 5 4 4 6 5 4 2
04/30 Write block 473 len 24576 data: 8 8 8 8 9 11
05/30 Flush
06/30 Flush
07/30 Write block 181 len 12288 data: 16 14 12
08/30 Write block 262 len 24576 data: 6 7 8 9 8 6
09/30 Write block 476 len 28672 data: 9 10 12 9 10 9 9
10/30 Read block 165 len 40960
11/30 Write block 373 len 36864 data: 10 12 14 14 13 13 12 8 6
12/30 Read block 535 len 4096
13/30 Read block 591 len 4096
14/30 Write block 548 len 32768 data: 8 11 10 13 11 8 10 9
15/30 Read block 412 len 4096
16/30 Write block 84 len 20480 data: 12 15 13 10 9
17/30 Write block 234 len 20480 data: 5 6 5 5 3
18/30 Read block 444 len 32768
19/30 Write block 258 len 36864 data: 6 5 7 8 7 8 9 10 9
20/30 Read block 51 len 16384
21/30 Flush
22/30 Flush
23/30 Write block 348 len 16384 data: 8 9 8 7
24/30 Write block 386 len 12288 data: 6 6 7
25/30 Write block 427 len 36864 data: 6 6 8 10 9 12 12 12 13
26/30 Write block 471 len 8192 data: 7 8
27/30 Read block 280 len 20480
28/30 Read block 549 len 4096
29/30 Read block 251 len 16384
30/30 Read block 537 len 24576
----------------------------------------------------------------
Crucible gen:100 GIO:true work queues: Upstairs:0 downstairs:8
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1022 Acked Write 4 Done Done Done false
1023 Acked Write 3 Done Done Done false
1024 Acked Write 9 Done Done Done false
1025 Acked Write 2 Done Done Done false
1026 Acked Read 5 Done Done Done false
1027 Acked Read 1 Done Done Done false
1028 Acked Read 4 Done Done Done false
1029 Acked Read 6 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 8 8 8 24
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1021 1021 1021
Downstairs last five completed: 1021 1020 1019 1018 1017
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:31.746507261Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1582}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 94 94 94 172 172 172 F F F
1 020-039 99 99 99 181 181 181 F F F
2 040-059 96 96 96 176 176 176 F F F
3 060-079 99 99 99 181 181 181 F F F
4 080-099 100 100 100 184 184 184 F F F
5 100-119 94 94 94 171 171 171 F F F
6 120-139 94 94 94 172 172 172 F F F
7 140-159 98 98 98 178 178 178 F F F
8 160-179 99 99 99 181 181 181 F F F
9 180-199 100 100 100 184 184 184 F F F
10 200-219 98 98 98 180 180 180 F F F
11 220-239 100 100 100 184 184 184 F F F
12 240-259 100 100 100 184 184 184 F F F
13 260-279 100 100 100 184 184 184 F F F
14 280-299 100 100 100 182 182 182 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 90 90 90 164 164 164 F F F
17 340-359 97 97 97 177 177 177 T T T
18 360-379 100 100 100 184 184 184 F F F
19 380-399 100 100 100 184 184 184 T T T
20 400-419 95 95 95 173 173 173 F F F
21 420-439 95 95 95 173 173 173 T T T
22 440-459 96 96 96 174 174 174 F F F
23 460-479 100 100 100 184 184 184 T T T
24 480-499 100 100 100 184 184 184 F F F
25 500-519 96 96 96 176 176 176 F F F
26 520-539 99 99 99 181 181 181 F F F
27 540-559 100 100 100 184 184 184 F F F
28 560-579 99 99 99 181 181 181 F F F
29 580-599 100 100 100 182 182 182 F F F
Max gen: 100, Max flush: 184
A Difference in extent metadata is expected here
On loop 50
Verifying data now
No change after verify
Loop: 50 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 94 94 94 172 172 172 F F F
1 020-039 99 99 99 181 181 181 F F F
2 040-059 96 96 96 176 176 176 F F F
3 060-079 99 99 99 181 181 181 F F F
4 080-099 100 100 100 184 184 184 F F F
5 100-119 94 94 94 171 171 171 F F F
6 120-139 94 94 94 172 172 172 F F F
7 140-159 98 98 98 178 178 178 F F F
8 160-179 99 99 99 181 181 181 F F F
9 180-199 100 100 100 184 184 184 F F F
10 200-219 98 98 98 180 180 180 F F F
11 220-239 100 100 100 184 184 184 F F F
12 240-259 100 100 100 184 184 184 F F F
13 260-279 100 100 100 184 184 184 F F F
14 280-299 100 100 100 182 182 182 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 90 90 90 164 164 164 F F F
17 340-359 101 101 101 185 185 185 F F F
18 360-379 100 100 100 184 184 184 F F F
19 380-399 101 101 101 185 185 185 F F F
20 400-419 95 95 95 173 173 173 F F F
21 420-439 101 101 101 185 185 185 F F F
22 440-459 96 96 96 174 174 174 F F F
23 460-479 101 101 101 185 185 185 F F F
24 480-499 100 100 100 184 184 184 F F F
25 500-519 96 96 96 176 176 176 F F F
26 520-539 99 99 99 181 181 181 F F F
27 540-559 100 100 100 184 184 184 F F F
28 560-579 99 99 99 181 181 181 F F F
29 580-599 100 100 100 182 182 182 F F F
Max gen: 101, Max flush: 185
Begin loop 51
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:35.991506434Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:36.034170214Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.03472565Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.066928918Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.067917446Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.067964357Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.067981906Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.067999615Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590}
{"msg":"Crucible cc35fd2b-5f0a-417f-85dc-6674e0d88663 has session id: c3b6933f-f58f-47bf-9ddc-85c09c96de27","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.068657248Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"Upstairs opts: Upstairs UUID: cc35fd2b-5f0a-417f-85dc-6674e0d88663, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.06870304Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"Crucible stats registered with UUID: cc35fd2b-5f0a-417f-85dc-6674e0d88663","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.068726715Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:36.068747323Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069174148Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"task":"crutest"}
{"msg":"The guest has requested activation with gen:102","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069208397Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590}
{"msg":"cc35fd2b-5f0a-417f-85dc-6674e0d88663 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069263723Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069289007Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"0","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069318958Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"0","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069334699Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"1","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069349779Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"1","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069363681Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"2","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069378212Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"2","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069391843Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"io task","client":"2","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069411061Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"io task","client":"0","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069432028Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"io task","client":"1","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"ds_connection connected from Ok(127.0.0.1:44006)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069650453Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"io task","client":"0","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"ds_connection connected from Ok(127.0.0.1:65043)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069786899Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"io task","client":"2","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"ds_connection connected from Ok(127.0.0.1:48732)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.069820088Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"io task","client":"1","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.070470765Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"1","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.07051113Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"2","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.070662537Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"0","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.071118244Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"1","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.071328754Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"2","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.071484737Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"0","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.071721481Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.071893834Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"[0]R flush_numbers[0..12]: [172, 181, 176, 181, 184, 171, 172, 178, 181, 184, 180, 184]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072150995Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"downstairs","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"[0]R generation[0..12]: [94, 99, 96, 99, 100, 94, 94, 98, 99, 100, 98, 100]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072180406Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"downstairs","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072195797Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"downstairs","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"[1]R flush_numbers[0..12]: [172, 181, 176, 181, 184, 171, 172, 178, 181, 184, 180, 184]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072209718Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"downstairs","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"[1]R generation[0..12]: [94, 99, 96, 99, 100, 94, 94, 98, 99, 100, 98, 100]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072224539Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"downstairs","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072238441Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"downstairs","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"[2]R flush_numbers[0..12]: [172, 181, 176, 181, 184, 171, 172, 178, 181, 184, 180, 184]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072251872Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"downstairs","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"[2]R generation[0..12]: [94, 99, 96, 99, 100, 94, 94, 98, 99, 100, 98, 100]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072265094Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"downstairs","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072278446Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"downstairs","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
Wait for a query_work_queue command to finish before sending IO
{"msg":"Max found gen is 102","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072300202Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"downstairs","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"Generation requested: 102 >= found:102","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072313534Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"downstairs","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"Next flush: 186","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072326106Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"downstairs","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072338579Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"":"downstairs","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072351291Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072363004Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"0","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072375805Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"1","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072388488Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"client":"2","":"downstairs","session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.07240132Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072413382Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"cc35fd2b-5f0a-417f-85dc-6674e0d88663 is now active with session: c3b6933f-f58f-47bf-9ddc-85c09c96de27","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072426005Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.072437577Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590,"session_id":"c3b6933f-f58f-47bf-9ddc-85c09c96de27"}
{"msg":"The guest has finished waiting for activation with:102","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:36.07244945Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 46 len 28672 data: 6 5 5 6 6 7 7
02/30 Read block 253 len 36864
03/30 Read block 400 len 24576
04/30 Write block 3 len 4096 data: 8
05/30 Write block 206 len 12288 data: 9 9 10
06/30 Write block 552 len 8192 data: 12 9
07/30 Read block 294 len 16384
08/30 Write block 243 len 20480 data: 6 7 7 9 11
09/30 Read block 90 len 8192
10/30 Read block 561 len 28672
11/30 Flush
12/30 Write block 457 len 36864 data: 7 7 7 8 7 9 8 7 6
13/30 Flush
14/30 Write block 98 len 40960 data: 8 8 10 11 10 10 13 14 12 8
15/30 Read block 292 len 4096
16/30 Write block 321 len 28672 data: 5 6 9 10 11 9 12
17/30 Read block 212 len 20480
18/30 Write block 134 len 16384 data: 7 6 6 7
19/30 Write block 535 len 20480 data: 12 9 8 7 7
20/30 Read block 471 len 4096
21/30 Flush
22/30 Write block 495 len 32768 data: 12 11 9 9 9 6 5 6
23/30 Flush
24/30 Write block 253 len 28672 data: 8 9 10 8 9 7 6
25/30 Write block 587 len 24576 data: 5 5 6 5 4 4
26/30 Read block 248 len 8192
27/30 Read block 62 len 8192
28/30 Read block 530 len 24576
29/30 Write block 108 len 12288 data: 9 9 9
30/30 Write block 241 len 40960 data: 4 5 7 8 8 10 12 12 10 7
----------------------------------------------------------------
Crucible gen:102 GIO:true work queues: Upstairs:0 downstairs:7
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1023 Acked Write 7 Done Done Done false
1024 Acked Write 6 Done Done Done false
1025 Acked Read 2 Done Done Done false
1026 Acked Read 2 Done Done Done false
1027 Acked Read 6 Done Done Done false
1028 Acked Write 3 Done Done Done false
1029 Acked Write 10 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 6 6 6 18
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1022 1022 1022
Downstairs last five completed: 1022 1021 1020 1019 1018
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:36.093719209Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1590}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 102 102 102 186 186 186 F F F
1 020-039 99 99 99 181 181 181 F F F
2 040-059 102 102 102 186 186 186 F F F
3 060-079 99 99 99 181 181 181 F F F
4 080-099 102 102 102 188 188 188 F F F
5 100-119 102 102 102 188 188 188 T T T
6 120-139 102 102 102 188 188 188 F F F
7 140-159 98 98 98 178 178 178 F F F
8 160-179 99 99 99 181 181 181 F F F
9 180-199 100 100 100 184 184 184 F F F
10 200-219 102 102 102 186 186 186 F F F
11 220-239 100 100 100 184 184 184 F F F
12 240-259 102 102 102 186 186 186 T T T
13 260-279 100 100 100 184 184 184 F F F
14 280-299 100 100 100 182 182 182 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 102 102 102 188 188 188 F F F
17 340-359 101 101 101 185 185 185 F F F
18 360-379 100 100 100 184 184 184 F F F
19 380-399 101 101 101 185 185 185 F F F
20 400-419 95 95 95 173 173 173 F F F
21 420-439 101 101 101 185 185 185 F F F
22 440-459 102 102 102 187 187 187 F F F
23 460-479 102 102 102 187 187 187 F F F
24 480-499 102 102 102 189 189 189 F F F
25 500-519 102 102 102 189 189 189 F F F
26 520-539 102 102 102 188 188 188 F F F
27 540-559 102 102 102 186 186 186 F F F
28 560-579 99 99 99 181 181 181 F F F
29 580-599 100 100 100 182 182 182 T T T
Max gen: 102, Max flush: 189
A Difference in extent metadata is expected here
On loop 51
Verifying data now
No change after verify
Loop: 51 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 102 102 102 186 186 186 F F F
1 020-039 99 99 99 181 181 181 F F F
2 040-059 102 102 102 186 186 186 F F F
3 060-079 99 99 99 181 181 181 F F F
4 080-099 102 102 102 188 188 188 F F F
5 100-119 103 103 103 190 190 190 F F F
6 120-139 102 102 102 188 188 188 F F F
7 140-159 98 98 98 178 178 178 F F F
8 160-179 99 99 99 181 181 181 F F F
9 180-199 100 100 100 184 184 184 F F F
10 200-219 102 102 102 186 186 186 F F F
11 220-239 100 100 100 184 184 184 F F F
12 240-259 103 103 103 190 190 190 F F F
13 260-279 100 100 100 184 184 184 F F F
14 280-299 100 100 100 182 182 182 F F F
15 300-319 89 89 89 163 163 163 F F F
16 320-339 102 102 102 188 188 188 F F F
17 340-359 101 101 101 185 185 185 F F F
18 360-379 100 100 100 184 184 184 F F F
19 380-399 101 101 101 185 185 185 F F F
20 400-419 95 95 95 173 173 173 F F F
21 420-439 101 101 101 185 185 185 F F F
22 440-459 102 102 102 187 187 187 F F F
23 460-479 102 102 102 187 187 187 F F F
24 480-499 102 102 102 189 189 189 F F F
25 500-519 102 102 102 189 189 189 F F F
26 520-539 102 102 102 188 188 188 F F F
27 540-559 102 102 102 186 186 186 F F F
28 560-579 99 99 99 181 181 181 F F F
29 580-599 103 103 103 190 190 190 F F F
Max gen: 103, Max flush: 190
Begin loop 52
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.366242607Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.411822192Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.41277776Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.412831667Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.412849496Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.412867584Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598}
{"msg":"Crucible 1edaab62-a2fb-4cf4-9f0e-7a26f2d61513 has session id: 0dc6b091-231e-46e2-a981-5ba2c30466c5","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.413457749Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"Upstairs opts: Upstairs UUID: 1edaab62-a2fb-4cf4-9f0e-7a26f2d61513, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.41350478Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"Crucible stats registered with UUID: 1edaab62-a2fb-4cf4-9f0e-7a26f2d61513","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.413533122Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:40.41355318Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414034442Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"task":"crutest"}
{"msg":"The guest has requested activation with gen:104","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414063654Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598}
{"msg":"1edaab62-a2fb-4cf4-9f0e-7a26f2d61513 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414131931Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414149471Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"0","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414186178Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"0","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414218948Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"1","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414245021Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"1","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414266558Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"2","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414289424Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"2","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414311091Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"io task","client":"2","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414334176Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"io task","client":"0","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414353344Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"io task","client":"1","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"ds_connection connected from Ok(127.0.0.1:65095)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414461497Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"io task","client":"1","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"ds_connection connected from Ok(127.0.0.1:41600)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414718248Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"io task","client":"0","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"ds_connection connected from Ok(127.0.0.1:46309)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.414762301Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"io task","client":"2","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.4153289Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"0","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.415370484Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"1","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.415400176Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"2","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416064135Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"0","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416094216Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"1","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416117022Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"2","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416708376Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.41673397Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"[0]R flush_numbers[0..12]: [186, 181, 186, 181, 188, 190, 188, 178, 181, 184, 186, 184]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416804057Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"downstairs","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"[0]R generation[0..12]: [102, 99, 102, 99, 102, 103, 102, 98, 99, 100, 102, 100]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416822655Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"downstairs","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416836876Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"downstairs","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"[1]R flush_numbers[0..12]: [186, 181, 186, 181, 188, 190, 188, 178, 181, 184, 186, 184]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416850498Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"downstairs","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"[1]R generation[0..12]: [102, 99, 102, 99, 102, 103, 102, 98, 99, 100, 102, 100]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416864159Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"downstairs","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416881089Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"downstairs","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"[2]R flush_numbers[0..12]: [186, 181, 186, 181, 188, 190, 188, 178, 181, 184, 186, 184]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.41689493Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"downstairs","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"[2]R generation[0..12]: [102, 99, 102, 99, 102, 103, 102, 98, 99, 100, 102, 100]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416908472Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"downstairs","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416922263Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"downstairs","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"Max found gen is 104","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416935115Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"downstairs","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"Generation requested: 104 >= found:104","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:40:40.416948107Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"downstairs","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"Next flush: 191","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416965767Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"downstairs","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.416979398Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"":"downstairs","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.41699232Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.417003973Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"0","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.417016895Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"1","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.417029427Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"client":"2","":"downstairs","session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.417042109Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.417053992Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"1edaab62-a2fb-4cf4-9f0e-7a26f2d61513 is now active with session: 0dc6b091-231e-46e2-a981-5ba2c30466c5","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.417066444Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.417079486Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598,"session_id":"0dc6b091-231e-46e2-a981-5ba2c30466c5"}
{"msg":"The guest has finished waiting for activation with:104","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:40.417091369Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 148 len 28672 data: 11 12 11 10 10 15 14
02/30 Read block 370 len 4096
03/30 Read block 553 len 16384
04/30 Read block 154 len 20480
05/30 Write block 254 len 36864 data: 10 11 9 10 8 7 8 9 8
06/30 Write block 347 len 24576 data: 6 9 10 9 8 7
07/30 Write block 44 len 24576 data: 6 7 7 6 6 7
08/30 Read block 202 len 4096
09/30 Flush
10/30 Write block 281 len 8192 data: 10 9
11/30 Read block 15 len 24576
12/30 Write block 575 len 28672 data: 8 7 7 5 5 6 6
13/30 Read block 573 len 36864
14/30 Write block 66 len 32768 data: 11 11 11 9 11 11 9 11
15/30 Write block 321 len 28672 data: 6 7 10 11 12 10 13
16/30 Read block 45 len 24576
17/30 Write block 214 len 4096 data: 7
18/30 Write block 415 len 36864 data: 3 5 7 8 10 10 7 7 9
19/30 Write block 314 len 8192 data: 10 9
20/30 Flush
21/30 Write block 560 len 12288 data: 7 5 4
22/30 Read block 430 len 36864
23/30 Read block 94 len 24576
24/30 Write block 27 len 28672 data: 8 8 8 11 9 8 8
25/30 Read block 9 len 24576
26/30 Write block 421 len 28672 data: 8 8 10 8 8 7 7
27/30 Read block 547 len 4096
28/30 Read block 432 len 40960
29/30 Write block 542 len 40960 data: 5 8 8 7 8 8 9 12 11 14
30/30 Write block 137 len 12288 data: 8 7 8
----------------------------------------------------------------
Crucible gen:104 GIO:true work queues: Upstairs:0 downstairs:10
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1020 Acked Write 3 Done Done Done false
1021 Acked Read 9 Done Done Done false
1022 Acked Read 6 Done Done Done false
1023 Acked Write 7 Done Done Done false
1024 Acked Read 6 Done Done Done false
1025 Acked Write 7 Done Done Done false
1026 Acked Read 1 Done Done Done false
1027 Acked Read 10 Done Done Done false
1028 Acked Write 10 Sent Sent Sent false
1029 Acked Write 3 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 8 8 8 24
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1019 1019 1019
Downstairs last five completed: 1019 1018 1017 1016 1015
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:40.437295467Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1598}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 102 102 102 186 186 186 F F F
1 020-039 99 99 99 181 181 181 T T T
2 040-059 104 104 104 191 191 191 F F F
3 060-079 104 104 104 192 192 192 F F F
4 080-099 102 102 102 188 188 188 F F F
5 100-119 103 103 103 190 190 190 F F F
6 120-139 102 102 102 188 188 188 T T T
7 140-159 104 104 104 191 191 191 F F F
8 160-179 99 99 99 181 181 181 F F F
9 180-199 100 100 100 184 184 184 F F F
10 200-219 104 104 104 192 192 192 F F F
11 220-239 100 100 100 184 184 184 F F F
12 240-259 104 104 104 191 191 191 F F F
13 260-279 104 104 104 191 191 191 F F F
14 280-299 104 104 104 192 192 192 F F F
15 300-319 104 104 104 192 192 192 F F F
16 320-339 104 104 104 192 192 192 F F F
17 340-359 104 104 104 191 191 191 F F F
18 360-379 100 100 100 184 184 184 F F F
19 380-399 101 101 101 185 185 185 F F F
20 400-419 104 104 104 192 192 192 F F F
21 420-439 104 104 104 192 192 192 T T T
22 440-459 102 102 102 187 187 187 F F F
23 460-479 102 102 102 187 187 187 F F F
24 480-499 102 102 102 189 189 189 F F F
25 500-519 102 102 102 189 189 189 F F F
26 520-539 102 102 102 188 188 188 F F F
27 540-559 102 102 102 186 186 186 T T T
28 560-579 104 104 104 192 192 192 T T T
29 580-599 104 104 104 192 192 192 F F F
Max gen: 104, Max flush: 192
A Difference in extent metadata is expected here
On loop 52
Verifying data now
No change after verify
Loop: 52 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 102 102 102 186 186 186 F F F
1 020-039 105 105 105 193 193 193 F F F
2 040-059 104 104 104 191 191 191 F F F
3 060-079 104 104 104 192 192 192 F F F
4 080-099 102 102 102 188 188 188 F F F
5 100-119 103 103 103 190 190 190 F F F
6 120-139 105 105 105 193 193 193 F F F
7 140-159 104 104 104 191 191 191 F F F
8 160-179 99 99 99 181 181 181 F F F
9 180-199 100 100 100 184 184 184 F F F
10 200-219 104 104 104 192 192 192 F F F
11 220-239 100 100 100 184 184 184 F F F
12 240-259 104 104 104 191 191 191 F F F
13 260-279 104 104 104 191 191 191 F F F
14 280-299 104 104 104 192 192 192 F F F
15 300-319 104 104 104 192 192 192 F F F
16 320-339 104 104 104 192 192 192 F F F
17 340-359 104 104 104 191 191 191 F F F
18 360-379 100 100 100 184 184 184 F F F
19 380-399 101 101 101 185 185 185 F F F
20 400-419 104 104 104 192 192 192 F F F
21 420-439 105 105 105 193 193 193 F F F
22 440-459 102 102 102 187 187 187 F F F
23 460-479 102 102 102 187 187 187 F F F
24 480-499 102 102 102 189 189 189 F F F
25 500-519 102 102 102 189 189 189 F F F
26 520-539 102 102 102 188 188 188 F F F
27 540-559 105 105 105 193 193 193 F F F
28 560-579 105 105 105 193 193 193 F F F
29 580-599 104 104 104 192 192 192 F F F
Max gen: 105, Max flush: 193
Begin loop 53
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.698858014Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:44.741505905Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.742152514Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.77398877Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.77487621Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.774920413Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.774938132Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.774955981Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606}
{"msg":"Crucible 8a36b9b1-0656-406b-8ddd-195d2353127c has session id: 4657bc14-a0c8-48be-895d-0f4c93fc1615","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.775654698Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"Upstairs opts: Upstairs UUID: 8a36b9b1-0656-406b-8ddd-195d2353127c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.775707985Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"Crucible stats registered with UUID: 8a36b9b1-0656-406b-8ddd-195d2353127c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.775733919Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:44.775750269Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.776167631Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"task":"crutest"}
{"msg":"The guest has requested activation with gen:106","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.776195213Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606}
{"msg":"8a36b9b1-0656-406b-8ddd-195d2353127c active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.776242984Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.776261362Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"0","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.776290045Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"0","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.77636124Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"1","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.776450256Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"1","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.7764756Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"2","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.776502173Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"2","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.776532425Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"io task","client":"2","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.77655669Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"io task","client":"0","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.77673402Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"io task","client":"1","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53445)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.776769198Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"io task","client":"2","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"ds_connection connected from Ok(127.0.0.1:45350)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.776788416Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"io task","client":"1","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"ds_connection connected from Ok(127.0.0.1:65004)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.776807704Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"io task","client":"0","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.777259215Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"1","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.777393611Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"2","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.777580106Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"0","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.777841824Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"1","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778023791Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"2","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778225126Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"0","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778472553Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778540691Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"[0]R flush_numbers[0..12]: [186, 193, 191, 192, 188, 190, 193, 191, 181, 184, 192, 184]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.77880048Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"downstairs","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"[0]R generation[0..12]: [102, 105, 104, 104, 102, 103, 105, 104, 99, 100, 104, 100]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778823036Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"downstairs","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778837667Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"downstairs","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"[1]R flush_numbers[0..12]: [186, 193, 191, 192, 188, 190, 193, 191, 181, 184, 192, 184]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778851609Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"downstairs","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"[1]R generation[0..12]: [102, 105, 104, 104, 102, 103, 105, 104, 99, 100, 104, 100]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.77886523Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"downstairs","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778878961Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"downstairs","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"[2]R flush_numbers[0..12]: [186, 193, 191, 192, 188, 190, 193, 191, 181, 184, 192, 184]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778892343Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"downstairs","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"[2]R generation[0..12]: [102, 105, 104, 104, 102, 103, 105, 104, 99, 100, 104, 100]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778905825Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"downstairs","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778919336Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"downstairs","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"Max found gen is 106","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778931829Z","hostname":"Wait for a query_work_queue command to finish before sending IO
w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"downstairs","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"Generation requested: 106 >= found:106","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778951696Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"downstairs","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"Next flush: 194","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778969166Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"downstairs","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.778982308Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"":"downstairs","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.77899538Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.779007032Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"0","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.779019954Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"1","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.779032986Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"client":"2","":"downstairs","session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.779045429Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.779057101Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"8a36b9b1-0656-406b-8ddd-195d2353127c is now active with session: 4657bc14-a0c8-48be-895d-0f4c93fc1615","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.779069483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.779081026Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606,"session_id":"4657bc14-a0c8-48be-895d-0f4c93fc1615"}
{"msg":"The guest has finished waiting for activation with:106","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:44.779092889Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 353 len 4096 data: 7
02/30 Read block 82 len 12288
03/30 Read block 90 len 36864
04/30 Read block 220 len 4096
05/30 Read block 327 len 36864
06/30 Write block 224 len 16384 data: 6 4 5 5
07/30 Read block 382 len 28672
08/30 Write block 60 len 36864 data: 7 7 8 8 8 11 12 12 12
09/30 Read block 555 len 40960
10/30 Write block 147 len 4096 data: 12
11/30 Read block 83 len 28672
12/30 Read block 524 len 36864
13/30 Write block 43 len 16384 data: 7 7 8 8
14/30 Flush
15/30 Write block 22 len 4096 data: 9
16/30 Read block 30 len 32768
17/30 Read block 444 len 4096
18/30 Read block 165 len 20480
19/30 Read block 179 len 8192
20/30 Write block 585 len 24576 data: 6 7 6 6 7 6
21/30 Read block 431 len 4096
22/30 Read block 384 len 8192
23/30 Write block 496 len 20480 data: 12 10 10 10 7
24/30 Read block 263 len 8192
25/30 Read block 126 len 4096
26/30 Write block 514 len 12288 data: 7 8 7
27/30 Read block 16 len 20480
28/30 Write block 148 len 24576 data: 12 13 12 11 11 16
29/30 Write block 74 len 36864 data: 10 9 6 6 8 7 9 10 11
30/30 Read block 477 len 16384
----------------------------------------------------------------
Crucible gen:106 GIO:true work queues: Upstairs:0 downstairs:16
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1014 Acked Write 1 Done Done Done false
1015 Acked Read 8 Done Done Done false
1016 Acked Read 1 Done Done Done false
1017 Acked Read 5 Done Done Done false
1018 Acked Read 2 Done Done Done false
1019 Acked Write 6 Done Done Done false
1020 Acked Read 1 Done Done Done false
1021 Acked Read 2 Done Done Done false
1022 Acked Write 5 Done Done Done false
1023 Acked Read 2 Done Done Done false
1024 Acked Read 1 Done Done Done false
1025 Acked Write 3 Done Done Done false
1026 Acked Read 5 Done Done Done false
1027 Acked Write 6 Done Done Done false
1028 Acked Write 9 Done Done Sent false
1029 Acked Read 4 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 2 2
Done 16 16 14 46
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1013 1013 1013
Downstairs last five completed: 1013 1012 1011 1010 1009
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:44.799478356Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1606}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 102 102 102 186 186 186 F F F
1 020-039 105 105 105 193 193 193 T T T
2 040-059 106 106 106 194 194 194 F F F
3 060-079 106 106 106 194 194 194 T T T
4 080-099 102 102 102 188 188 188 T T T
5 100-119 103 103 103 190 190 190 F F F
6 120-139 105 105 105 193 193 193 F F F
7 140-159 106 106 106 194 194 194 T T T
8 160-179 99 99 99 181 181 181 F F F
9 180-199 100 100 100 184 184 184 F F F
10 200-219 104 104 104 192 192 192 F F F
11 220-239 106 106 106 194 194 194 F F F
12 240-259 104 104 104 191 191 191 F F F
13 260-279 104 104 104 191 191 191 F F F
14 280-299 104 104 104 192 192 192 F F F
15 300-319 104 104 104 192 192 192 F F F
16 320-339 104 104 104 192 192 192 F F F
17 340-359 106 106 106 194 194 194 F F F
18 360-379 100 100 100 184 184 184 F F F
19 380-399 101 101 101 185 185 185 F F F
20 400-419 104 104 104 192 192 192 F F F
21 420-439 105 105 105 193 193 193 F F F
22 440-459 102 102 102 187 187 187 F F F
23 460-479 102 102 102 187 187 187 F F F
24 480-499 102 102 102 189 189 189 T T T
25 500-519 102 102 102 189 189 189 T T T
26 520-539 102 102 102 188 188 188 F F F
27 540-559 105 105 105 193 193 193 F F F
28 560-579 105 105 105 193 193 193 F F F
29 580-599 104 104 104 192 192 192 T T T
Max gen: 106, Max flush: 194
A Difference in extent metadata is expected here
On loop 53
Verifying data now
No change after verify
Loop: 53 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 102 102 102 186 186 186 F F F
1 020-039 107 107 107 195 195 195 F F F
2 040-059 106 106 106 194 194 194 F F F
3 060-079 107 107 107 195 195 195 F F F
4 080-099 107 107 107 195 195 195 F F F
5 100-119 103 103 103 190 190 190 F F F
6 120-139 105 105 105 193 193 193 F F F
7 140-159 107 107 107 195 195 195 F F F
8 160-179 99 99 99 181 181 181 F F F
9 180-199 100 100 100 184 184 184 F F F
10 200-219 104 104 104 192 192 192 F F F
11 220-239 106 106 106 194 194 194 F F F
12 240-259 104 104 104 191 191 191 F F F
13 260-279 104 104 104 191 191 191 F F F
14 280-299 104 104 104 192 192 192 F F F
15 300-319 104 104 104 192 192 192 F F F
16 320-339 104 104 104 192 192 192 F F F
17 340-359 106 106 106 194 194 194 F F F
18 360-379 100 100 100 184 184 184 F F F
19 380-399 101 101 101 185 185 185 F F F
20 400-419 104 104 104 192 192 192 F F F
21 420-439 105 105 105 193 193 193 F F F
22 440-459 102 102 102 187 187 187 F F F
23 460-479 102 102 102 187 187 187 F F F
24 480-499 107 107 107 195 195 195 F F F
25 500-519 107 107 107 195 195 195 F F F
26 520-539 102 102 102 188 188 188 F F F
27 540-559 105 105 105 193 193 193 F F F
28 560-579 105 105 105 193 193 193 F F F
29 580-599 107 107 107 195 195 195 F F F
Max gen: 107, Max flush: 195
Begin loop 54
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.097653629Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:49.142581577Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.143315953Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.175997615Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.176971561Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.177024289Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.177040529Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.17705452Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614}
{"msg":"Crucible 149a28d2-bd3b-4689-ba56-e5659eb14e13 has session id: e2faef9f-602a-4c89-93a0-36bdb8a648a7","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.177586401Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"Upstairs opts: Upstairs UUID: 149a28d2-bd3b-4689-ba56-e5659eb14e13, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.177629484Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"Crucible stats registered with UUID: 149a28d2-bd3b-4689-ba56-e5659eb14e13","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.177658186Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:49.177678544Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178066184Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"task":"crutest"}
{"msg":"The guest has requested activation with gen:108","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178090299Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614}
{"msg":"149a28d2-bd3b-4689-ba56-e5659eb14e13 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178153789Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178179993Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"0","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178205907Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"0","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178220358Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"1","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178234639Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"1","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178247911Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"2","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178261933Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"2","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178283379Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"io task","client":"2","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178307594Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"io task","client":"0","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.17833088Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"io task","client":"1","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57891)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178391652Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"io task","client":"2","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"ds_connection connected from Ok(127.0.0.1:59644)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178691547Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"io task","client":"1","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"ds_connection connected from Ok(127.0.0.1:58071)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.178737728Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"io task","client":"0","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.179259155Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"1","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.179313341Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"2","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.179625258Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"0","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.180098125Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"1","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.180137831Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"2","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.18039697Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"0","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.18070377Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.18080094Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"[0]R flush_numbers[0..12]: [186, 195, 194, 195, 195, 190, 193, 195, 181, 184, 192, 194]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.180918547Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"downstairs","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"[0]R generation[0..12]: [102, 107, 106, 107, 107, 103, 105, 107, 99, 100, 104, 106]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.180939924Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"downstairs","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.180954585Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"downstairs","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"[1]R flush_numbers[0..12]: [186, 195, 194, 195, 195, 190, 193, 195, 181, 184, 192, 194]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.180968057Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"downstairs","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"[1]R generation[0..12]: [102, 107, 106, 107, 107, 103, 105, 107, 99, 100, 104, 106]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.180984157Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"downstairs","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181006033Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"downstairs","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"[2]R flush_numbers[0..12]: [186, 195, 194, 195, 195, 190, 193, 195, 181, 184, 192, 194]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181025061Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"downstairs","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"[2]R generation[0..12]: [102, 107, 106, 107, 107, 103, 105, 107, 99, 100, 104, 106]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181039033Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"downstairs","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181052434Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"downstairs","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"Max found gen is 108","Wait for a query_work_queue command to finish before sending IO
v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181071912Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"downstairs","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"Generation requested: 108 >= found:108","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181084705Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"downstairs","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"Next flush: 196","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181097057Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"downstairs","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181109489Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"":"downstairs","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181122401Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181133914Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"0","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181147066Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"1","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181159908Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"client":"2","":"downstairs","session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.18117286Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181187401Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"149a28d2-bd3b-4689-ba56-e5659eb14e13 is now active with session: e2faef9f-602a-4c89-93a0-36bdb8a648a7","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181200133Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181211576Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614,"session_id":"e2faef9f-602a-4c89-93a0-36bdb8a648a7"}
{"msg":"The guest has finished waiting for activation with:108","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:49.181223309Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 93 len 8192 data: 13 10
02/30 Read block 167 len 32768
03/30 Read block 157 len 16384
04/30 Read block 506 len 8192
05/30 Read block 194 len 28672
06/30 Write block 211 len 28672 data: 7 8 8 8 9 7 7
07/30 Flush
08/30 Write block 119 len 20480 data: 13 10 10 9 6
09/30 Read block 406 len 16384
10/30 Read block 97 len 16384
11/30 Write block 376 len 24576 data: 15 14 14 13 9 7
12/30 Write block 413 len 24576 data: 5 5 4 6 8 9
13/30 Read block 553 len 24576
14/30 Write block 323 len 40960 data: 11 12 13 11 14 9 8 7 6 7
15/30 Write block 155 len 20480 data: 8 9 8 8 7
16/30 Read block 518 len 16384
17/30 Write block 522 len 36864 data: 10 9 10 9 8 11 13 13 13
18/30 Read block 291 len 28672
19/30 Read block 527 len 20480
20/30 Write block 135 len 40960 data: 7 7 9 8 9 8 9 10 11 11
21/30 Read block 74 len 28672
22/30 Read block 141 len 8192
23/30 Write block 462 len 20480 data: 10 9 8 7 6
24/30 Write block 207 len 20480 data: 10 11 9 6 8
25/30 Flush
26/30 Flush
27/30 Write block 490 len 40960 data: 9 10 12 10 8 13 13 11 11 11
28/30 Write block 360 len 20480 data: 10 8 7 6 5
29/30 Write block 85 len 40960 data: 16 14 11 10 8 11 10 13 14 11
30/30 Read block 8 len 28672
----------------------------------------------------------------
Crucible gen:108 GIO:true work queues: Upstairs:0 downstairs:4
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1026 Acked Write 10 Done Done Done false
1027 Acked Write 5 Done Done Done false
1028 Acked Write 10 Sent Done Done false
1029 Acked Read 7 Sent Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 0 1 3
Done 2 4 3 9
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1025 1025 1025
Downstairs last five completed: 1025 1024 1023 1022 1021
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:49.202581902Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1614}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 102 102 102 186 186 186 F F F
1 020-039 107 107 107 195 195 195 F F F
2 040-059 106 106 106 194 194 194 F F F
3 060-079 107 107 107 195 195 195 F F F
4 080-099 108 108 108 196 196 196 T T T
5 100-119 108 108 108 197 197 197 F F F
6 120-139 108 108 108 197 197 197 F F F
7 140-159 108 108 108 197 197 197 F F F
8 160-179 99 99 99 181 181 181 F F F
9 180-199 100 100 100 184 184 184 F F F
10 200-219 108 108 108 197 197 197 F F F
11 220-239 106 106 106 194 194 194 F F F
12 240-259 104 104 104 191 191 191 F F F
13 260-279 104 104 104 191 191 191 F F F
14 280-299 104 104 104 192 192 192 F F F
15 300-319 104 104 104 192 192 192 F F F
16 320-339 108 108 108 197 197 197 F F F
17 340-359 106 106 106 194 194 194 F F F
18 360-379 108 108 108 197 197 197 T T T
19 380-399 108 108 108 197 197 197 F F F
20 400-419 108 108 108 197 197 197 F F F
21 420-439 105 105 105 193 193 193 F F F
22 440-459 102 102 102 187 187 187 F F F
23 460-479 108 108 108 197 197 197 F F F
24 480-499 107 107 107 195 195 195 T T T
25 500-519 107 107 107 195 195 195 F F F
26 520-539 108 108 108 197 197 197 F F F
27 540-559 105 105 105 193 193 193 F F F
28 560-579 105 105 105 193 193 193 F F F
29 580-599 107 107 107 195 195 195 F F F
Max gen: 108, Max flush: 197
A Difference in extent metadata is expected here
On loop 54
Verifying data now
No change after verify
Loop: 54 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 102 102 102 186 186 186 F F F
1 020-039 107 107 107 195 195 195 F F F
2 040-059 106 106 106 194 194 194 F F F
3 060-079 107 107 107 195 195 195 F F F
4 080-099 109 109 109 198 198 198 F F F
5 100-119 108 108 108 197 197 197 F F F
6 120-139 108 108 108 197 197 197 F F F
7 140-159 108 108 108 197 197 197 F F F
8 160-179 99 99 99 181 181 181 F F F
9 180-199 100 100 100 184 184 184 F F F
10 200-219 108 108 108 197 197 197 F F F
11 220-239 106 106 106 194 194 194 F F F
12 240-259 104 104 104 191 191 191 F F F
13 260-279 104 104 104 191 191 191 F F F
14 280-299 104 104 104 192 192 192 F F F
15 300-319 104 104 104 192 192 192 F F F
16 320-339 108 108 108 197 197 197 F F F
17 340-359 106 106 106 194 194 194 F F F
18 360-379 109 109 109 198 198 198 F F F
19 380-399 108 108 108 197 197 197 F F F
20 400-419 108 108 108 197 197 197 F F F
21 420-439 105 105 105 193 193 193 F F F
22 440-459 102 102 102 187 187 187 F F F
23 460-479 108 108 108 197 197 197 F F F
24 480-499 109 109 109 198 198 198 F F F
25 500-519 107 107 107 195 195 195 F F F
26 520-539 108 108 108 197 197 197 F F F
27 540-559 105 105 105 193 193 193 F F F
28 560-579 105 105 105 193 193 193 F F F
29 580-599 107 107 107 195 195 195 F F F
Max gen: 109, Max flush: 198
Begin loop 55
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.478652205Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:53.521752175Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.522236306Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.553715602Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.554600005Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.554640579Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.55465609Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.554679325Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622}
{"msg":"Crucible 67668339-be4f-447c-a54d-df7659c6c537 has session id: 63a7670b-818c-4166-ba18-c1b6bd24963e","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.55539861Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"Upstairs opts: Upstairs UUID: 67668339-be4f-447c-a54d-df7659c6c537, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.555448199Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"Crucible stats registered with UUID: 67668339-be4f-447c-a54d-df7659c6c537","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.555480129Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:53.555508542Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.555938476Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"task":"crutest"}
{"msg":"The guest has requested activation with gen:110","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.555970246Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622}
{"msg":"67668339-be4f-447c-a54d-df7659c6c537 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.556049097Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.556069804Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"0","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.556102974Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"0","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.556120363Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"1","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.556152103Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"1","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.556229975Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"2","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.556280973Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"2","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.556314772Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"io task","client":"2","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.556348731Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"io task","client":"0","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.556384469Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"io task","client":"1","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"ds_connection connected from Ok(127.0.0.1:61351)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.556418978Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"io task","client":"0","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"ds_connection connected from Ok(127.0.0.1:60095)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.556457274Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"io task","client":"1","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"ds_connection connected from Ok(127.0.0.1:46032)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.556493882Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"io task","client":"2","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.557276387Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"1","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.557447891Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"0","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.557500558Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"2","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.558041074Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"1","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.55817658Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"2","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.55822403Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"0","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.558670294Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.558829735Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"[0]R flush_numbers[0..12]: [186, 195, 194, 195, 198, 197, 197, 197, 181, 184, 197, 194]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.558901921Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"downstairs","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"[0]R generation[0..12]: [102, 107, 106, 107, 109, 108, 108, 108, 99, 100, 108, 106]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.558923617Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"downstairs","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.558939127Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"downstairs","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"[1]R flush_numbers[0..12]: [186, 195, 194, 195, 198, 197, 197, 197, 181, 184, 197, 194]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.558953009Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"downstairs","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"[1]R generation[0..12]: [102, 107, 106, 107, 109, 108, 108, 108, 99, 100, 108, 106]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.55896645Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"downstairs","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.558980142Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"downstairs","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"[2]R flush_numbers[0..12]: [186, 195, 194, 195, 198, 197, 197, 197, 181, 184, 197, 194]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.558993714Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"downstairs","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"[2]R generation[0..12]: [102, 107, 106, 107, 109, 108, 108, 108, 99, 100, 108, 106]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.559007255Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"downstairs","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.559020747Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"downstairs","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"Max found gen is 110","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.559033219Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"downstairs","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"Generation requested: 110 >= found:110","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.559045961Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"downstairs","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"Next flush: 199","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:40:53.55906331Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"downstairs","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.559079111Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"":"downstairs","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.559091913Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.559103466Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"0","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.559134656Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"1","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.559147568Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"client":"2","":"downstairs","session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.55916039Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.559172403Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"67668339-be4f-447c-a54d-df7659c6c537 is now active with session: 63a7670b-818c-4166-ba18-c1b6bd24963e","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.559184785Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.559196308Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622,"session_id":"63a7670b-818c-4166-ba18-c1b6bd24963e"}
{"msg":"The guest has finished waiting for activation with:110","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:53.559208121Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 401 len 16384 data: 7 8 8 8
02/30 Write block 165 len 28672 data: 12 13 11 9 6 9 9
03/30 Flush
04/30 Flush
05/30 Write block 191 len 24576 data: 8 7 8 8 11 9
06/30 Write block 588 len 36864 data: 7 8 7 5 5 6 5 5 7
07/30 Read block 532 len 28672
08/30 Read block 50 len 12288
09/30 Read block 321 len 40960
10/30 Read block 378 len 24576
11/30 Read block 592 len 28672
12/30 Read block 559 len 40960
13/30 Write block 60 len 24576 data: 8 8 9 9 9 12
14/30 Read block 503 len 8192
15/30 Flush
16/30 Write block 279 len 12288 data: 12 11 11
17/30 Write block 504 len 8192 data: 5 5
18/30 Read block 349 len 28672
19/30 Write block 554 len 28672 data: 11 10 11 10 9 10 8
20/30 Read block 524 len 28672
21/30 Read block 194 len 12288
22/30 Write block 350 len 20480 data: 10 9 8 8 8
23/30 Read block 2 len 4096
24/30 Read block 460 len 4096
25/30 Write block 342 len 40960 data: 10 9 11 9 8 7 10 11 11 10
26/30 Write block 289 len 16384 data: 7 7 9 10
27/30 Write block 231 len 12288 data: 7 4 4
28/30 Read block 29 len 12288
29/30 Write block 516 len 28672 data: 8 6 6 7 7 10 11
30/30 Read block 485 len 40960
----------------------------------------------------------------
Crucible gen:110 GIO:true work queues: Upstairs:0 downstairs:15
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1015 Acked Write 3 Done Done Done false
1016 Acked Write 2 Done Done Done false
1017 Acked Read 7 Done Done Done false
1018 Acked Write 7 Done Done Done false
1019 Acked Read 7 Done Done Done false
1020 Acked Read 3 Done Done Done false
1021 Acked Write 5 Done Done Done false
1022 Acked Read 1 Done Done Done false
1023 Acked Read 1 Done Done Done false
1024 Acked Write 10 Done Done Done false
1025 Acked Write 4 Done Done Done false
1026 Acked Write 3 Done Done Done false
1027 Acked Read 3 Done Done Done false
1028 Acked Write 7 Done Done Done false
1029 Acked Read 10 Sent Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 0 1 2
Done 14 15 14 43
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1014 1014 1014
Downstairs last five completed: 1014 1013 1012 1011 1010
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:53.580619812Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1622}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 102 102 102 186 186 186 F F F
1 020-039 107 107 107 195 195 195 F F F
2 040-059 106 106 106 194 194 194 F F F
3 060-079 110 110 110 201 201 201 F F F
4 080-099 109 109 109 198 198 198 F F F
5 100-119 108 108 108 197 197 197 F F F
6 120-139 108 108 108 197 197 197 F F F
7 140-159 108 108 108 197 197 197 F F F
8 160-179 110 110 110 199 199 199 F F F
9 180-199 110 110 110 201 201 201 F F F
10 200-219 108 108 108 197 197 197 F F F
11 220-239 106 106 106 194 194 194 T T T
12 240-259 104 104 104 191 191 191 F F F
13 260-279 104 104 104 191 191 191 T T T
14 280-299 104 104 104 192 192 192 T T T
15 300-319 104 104 104 192 192 192 F F F
16 320-339 108 108 108 197 197 197 F F F
17 340-359 106 106 106 194 194 194 T T T
18 360-379 109 109 109 198 198 198 F F F
19 380-399 108 108 108 197 197 197 F F F
20 400-419 110 110 110 199 199 199 F F F
21 420-439 105 105 105 193 193 193 F F F
22 440-459 102 102 102 187 187 187 F F F
23 460-479 108 108 108 197 197 197 F F F
24 480-499 109 109 109 198 198 198 F F F
25 500-519 107 107 107 195 195 195 T T T
26 520-539 108 108 108 197 197 197 T T T
27 540-559 105 105 105 193 193 193 T T T
28 560-579 105 105 105 193 193 193 T T T
29 580-599 110 110 110 201 201 201 F F F
Max gen: 110, Max flush: 201
A Difference in extent metadata is expected here
On loop 55
Verifying data now
No change after verify
Loop: 55 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 102 102 102 186 186 186 F F F
1 020-039 107 107 107 195 195 195 F F F
2 040-059 106 106 106 194 194 194 F F F
3 060-079 110 110 110 201 201 201 F F F
4 080-099 109 109 109 198 198 198 F F F
5 100-119 108 108 108 197 197 197 F F F
6 120-139 108 108 108 197 197 197 F F F
7 140-159 108 108 108 197 197 197 F F F
8 160-179 110 110 110 199 199 199 F F F
9 180-199 110 110 110 201 201 201 F F F
10 200-219 108 108 108 197 197 197 F F F
11 220-239 111 111 111 202 202 202 F F F
12 240-259 104 104 104 191 191 191 F F F
13 260-279 111 111 111 202 202 202 F F F
14 280-299 111 111 111 202 202 202 F F F
15 300-319 104 104 104 192 192 192 F F F
16 320-339 108 108 108 197 197 197 F F F
17 340-359 111 111 111 202 202 202 F F F
18 360-379 109 109 109 198 198 198 F F F
19 380-399 108 108 108 197 197 197 F F F
20 400-419 110 110 110 199 199 199 F F F
21 420-439 105 105 105 193 193 193 F F F
22 440-459 102 102 102 187 187 187 F F F
23 460-479 108 108 108 197 197 197 F F F
24 480-499 109 109 109 198 198 198 F F F
25 500-519 111 111 111 202 202 202 F F F
26 520-539 111 111 111 202 202 202 F F F
27 540-559 111 111 111 202 202 202 F F F
28 560-579 111 111 111 202 202 202 F F F
29 580-599 110 110 110 201 201 201 F F F
Max gen: 111, Max flush: 202
Begin loop 56
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.879129198Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:57.923126812Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.923767446Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.956776725Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.957770729Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.957815162Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.957832171Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.95785038Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630}
{"msg":"Crucible 57b491a9-fd03-4970-a25b-c52f81af7c3e has session id: e426c4ec-dbc3-44f2-8b91-888e0a6031cc","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.958517537Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"Upstairs opts: Upstairs UUID: 57b491a9-fd03-4970-a25b-c52f81af7c3e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.958558401Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"Crucible stats registered with UUID: 57b491a9-fd03-4970-a25b-c52f81af7c3e","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.958578139Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:57.958591551Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.959068965Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"task":"crutest"}
{"msg":"The guest has requested activation with gen:112","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.959094549Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630}
{"msg":"57b491a9-fd03-4970-a25b-c52f81af7c3e active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.959181675Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.959199435Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"0","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.95922372Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"0","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.959237701Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"1","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.959252312Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"1","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.959266043Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"2","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.959280544Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"2","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.959295165Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"io task","client":"2","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.959376105Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"io task","client":"0","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.959424275Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"io task","client":"1","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57597)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.959463901Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"io task","client":"2","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49466)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.95949784Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"io task","client":"1","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50499)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.959726688Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"io task","client":"0","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.960324638Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"1","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.960367791Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"2","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.960649017Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"0","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.961050258Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"1","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.961127001Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"2","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.961383302Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"0","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.961713818Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.961736703Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"[0]R flush_numbers[0..12]: [186, 195, 194, 201, 198, 197, 197, 197, 199, 201, 197, 202]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.961969839Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"downstairs","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"[0]R generation[0..12]: [102, 107, 106, 110, 109, 108, 108, 108, 110, 110, 108, 111]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962007955Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"downstairs","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962034589Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"downstairs","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"[1]R flush_numbers[0..12]: [186, 195, 194, 201, 198, 197, 197, 197, 199, 201, 197, 202]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962059903Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"downstairs","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"[1]R generation[0..12]: [102, 107, 106, 110, 109, 108, 108, 108, 110, 110, 108, 111]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962089065Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"downstairs","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.96211299Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"downstairs","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"[2]R flush_numbers[0..12]: [186, 195, 194, 201, 198, 197, 197, 197, 199, 201, 197, 202]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962139144Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"downstairs","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{Wait for a query_work_queue command to finish before sending IO
"msg":"[2]R generation[0..12]: [102, 107, 106, 110, 109, 108, 108, 108, 110, 110, 108, 111]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962173003Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"downstairs","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962204124Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"downstairs","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"Max found gen is 112","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962231577Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"downstairs","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"Generation requested: 112 >= found:112","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962257701Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"downstairs","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"Next flush: 203","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962283885Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"downstairs","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962309738Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"":"downstairs","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962335742Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962362916Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"0","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.96238787Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"1","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962408737Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"client":"2","":"downstairs","session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962422109Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962438199Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"57b491a9-fd03-4970-a25b-c52f81af7c3e is now active with session: e426c4ec-dbc3-44f2-8b91-888e0a6031cc","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962451061Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962462864Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630,"session_id":"e426c4ec-dbc3-44f2-8b91-888e0a6031cc"}
{"msg":"The guest has finished waiting for activation with:112","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:40:57.962474786Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 21 len 24576 data: 7 10 11 12 10 10
02/30 Write block 586 len 16384 data: 8 7 8 9
03/30 Write block 486 len 32768 data: 9 12 11 8 10 11 13 11
04/30 Read block 437 len 32768
05/30 Read block 351 len 24576
06/30 Read block 225 len 28672
07/30 Write block 401 len 28672 data: 8 9 9 9 8 7 6
08/30 Read block 593 len 24576
09/30 Flush
10/30 Write block 23 len 28672 data: 12 13 11 11 9 9 9
11/30 Write block 223 len 32768 data: 7 7 5 6 6 6 5 7
12/30 Read block 103 len 40960
13/30 Read block 119 len 36864
14/30 Write block 357 len 32768 data: 10 12 12 11 9 8 7 6
15/30 Write block 128 len 8192 data: 4 5
16/30 Read block 156 len 36864
17/30 Write block 138 len 20480 data: 9 10 9 10 11
18/30 Read block 286 len 24576
19/30 Flush
20/30 Write block 296 len 16384 data: 11 10 8 9
21/30 Write block 556 len 28672 data: 12 11 10 11 9 6 5
22/30 Write block 405 len 28672 data: 9 8 7 6 7 6 6
23/30 Write block 240 len 20480 data: 4 5 6 8 9
24/30 Write block 63 len 4096 data: 10
25/30 Flush
26/30 Write block 10 len 8192 data: 9 7
27/30 Read block 463 len 24576
28/30 Write block 371 len 16384 data: 11 10 11 13
29/30 Read block 330 len 24576
30/30 Write block 138 len 24576 data: 10 11 10 11 12 12
----------------------------------------------------------------
Crucible gen:112 GIO:true work queues: Upstairs:0 downstairs:5
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1025 Acked Write 2 Done Done Done false
1026 Acked Read 6 Done Done Done false
1027 Acked Write 4 Done Done Done false
1028 Acked Read 6 Done Done Done false
1029 Acked Write 6 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 4 4 4 12
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1024 1024 1024
Downstairs last five completed: 1024 1023 1022 1021 1020
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:40:57.982245173Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1630}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 102 102 102 186 186 186 T T T
1 020-039 112 112 112 204 204 204 F F F
2 040-059 106 106 106 194 194 194 F F F
3 060-079 112 112 112 205 205 205 F F F
4 080-099 109 109 109 198 198 198 F F F
5 100-119 108 108 108 197 197 197 F F F
6 120-139 112 112 112 204 204 204 T T T
7 140-159 112 112 112 204 204 204 T T T
8 160-179 110 110 110 199 199 199 F F F
9 180-199 110 110 110 201 201 201 F F F
10 200-219 108 108 108 197 197 197 F F F
11 220-239 112 112 112 204 204 204 F F F
12 240-259 112 112 112 205 205 205 F F F
13 260-279 111 111 111 202 202 202 F F F
14 280-299 112 112 112 205 205 205 F F F
15 300-319 104 104 104 192 192 192 F F F
16 320-339 108 108 108 197 197 197 F F F
17 340-359 112 112 112 204 204 204 F F F
18 360-379 112 112 112 204 204 204 T T T
19 380-399 108 108 108 197 197 197 F F F
20 400-419 112 112 112 205 205 205 F F F
21 420-439 105 105 105 193 193 193 F F F
22 440-459 102 102 102 187 187 187 F F F
23 460-479 108 108 108 197 197 197 F F F
24 480-499 112 112 112 203 203 203 F F F
25 500-519 111 111 111 202 202 202 F F F
26 520-539 111 111 111 202 202 202 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 112 112 112 205 205 205 F F F
29 580-599 112 112 112 203 203 203 F F F
Max gen: 112, Max flush: 205
A Difference in extent metadata is expected here
On loop 56
Verifying data now
No change after verify
Loop: 56 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 113 113 113 206 206 206 F F F
1 020-039 112 112 112 204 204 204 F F F
2 040-059 106 106 106 194 194 194 F F F
3 060-079 112 112 112 205 205 205 F F F
4 080-099 109 109 109 198 198 198 F F F
5 100-119 108 108 108 197 197 197 F F F
6 120-139 113 113 113 206 206 206 F F F
7 140-159 113 113 113 206 206 206 F F F
8 160-179 110 110 110 199 199 199 F F F
9 180-199 110 110 110 201 201 201 F F F
10 200-219 108 108 108 197 197 197 F F F
11 220-239 112 112 112 204 204 204 F F F
12 240-259 112 112 112 205 205 205 F F F
13 260-279 111 111 111 202 202 202 F F F
14 280-299 112 112 112 205 205 205 F F F
15 300-319 104 104 104 192 192 192 F F F
16 320-339 108 108 108 197 197 197 F F F
17 340-359 112 112 112 204 204 204 F F F
18 360-379 113 113 113 206 206 206 F F F
19 380-399 108 108 108 197 197 197 F F F
20 400-419 112 112 112 205 205 205 F F F
21 420-439 105 105 105 193 193 193 F F F
22 440-459 102 102 102 187 187 187 F F F
23 460-479 108 108 108 197 197 197 F F F
24 480-499 112 112 112 203 203 203 F F F
25 500-519 111 111 111 202 202 202 F F F
26 520-539 111 111 111 202 202 202 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 112 112 112 205 205 205 F F F
29 580-599 112 112 112 203 203 203 F F F
Max gen: 113, Max flush: 206
Begin loop 57
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.263939212Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.309049188Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.310016898Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.310063799Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.310082837Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.310097988Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638}
{"msg":"Crucible 5cdb750b-1e60-4ec7-a376-cfa992bc3a95 has session id: a59245d0-26ab-4c57-8535-9311a373b203","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.310700655Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"Upstairs opts: Upstairs UUID: 5cdb750b-1e60-4ec7-a376-cfa992bc3a95, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.310750854Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"Crucible stats registered with UUID: 5cdb750b-1e60-4ec7-a376-cfa992bc3a95","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.310779536Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:02.310802682Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311426096Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"task":"crutest"}
{"msg":"The guest has requested activation with gen:114","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311459545Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638}
{"msg":"5cdb750b-1e60-4ec7-a376-cfa992bc3a95 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311527263Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311556035Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"0","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311583878Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"0","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311599578Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"1","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311614329Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"1","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311636395Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"2","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311663259Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"2","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311689453Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"io task","client":"2","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311713598Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"io task","client":"0","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311737923Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"io task","client":"1","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52342)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311784794Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"io task","client":"2","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"ds_connection connected from Ok(127.0.0.1:37236)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.311821631Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"io task","client":"0","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39747)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.31185568Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"io task","client":"1","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.312592973Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"0","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.312629011Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"2","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.312882754Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"1","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.31345447Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"2","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.313488559Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"1","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.313571117Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"0","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314098241Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314127123Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"[0]R flush_numbers[0..12]: [206, 204, 194, 205, 198, 197, 206, 206, 199, 201, 197, 204]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314140935Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"downstairs","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"[0]R generation[0..12]: [113, 112, 106, 112, 109, 108, 113, 113, 110, 110, 108, 112]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314155965Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"downstairs","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314170217Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"downstairs","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"[1]R flush_numbers[0..12]: [206, 204, 194, 205, 198, 197, 206, 206, 199, 201, 197, 204]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314184038Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"downstairs","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"[1]R generation[0..12]: [113, 112, 106, 112, 109, 108, 113, 113, 110, 110, 108, 112]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314197999Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"downstairs","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314211841Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"downstairs","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"[2]R flush_numbers[0..12]: [206, 204, 194, 205, 198, 197, 206, 206, 199, 201, 197, 204]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314225132Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"downstairs","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"[2]R generation[0..12]: [113, 112, 106, 112, 109, 108, 113, 113, 110, 110, 108, 112]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314238424Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"downstairs","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314251876Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"downstairs","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"Max found gen is 114","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314264648Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"downstairs","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"Generation requested: 114 >= found:114","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:41:02.31427782Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"downstairs","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"Next flush: 207","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314296008Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"downstairs","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314308631Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"":"downstairs","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314324701Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314336673Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"0","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314350605Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"1","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314363587Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"client":"2","":"downstairs","session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314376838Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314388651Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"5cdb750b-1e60-4ec7-a376-cfa992bc3a95 is now active with session: a59245d0-26ab-4c57-8535-9311a373b203","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314400974Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:02.314412456Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638,"session_id":"a59245d0-26ab-4c57-8535-9311a373b203"}
{"msg":"The guest has finished waiting for activation with:114","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
,"time":"2025-01-27T19:41:02.314424309Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 336 len 12288 data: 5 6 7
02/30 Write block 343 len 40960 data: 10 12 10 9 8 11 12 12 11 9
03/30 Write block 62 len 40960 data: 10 11 10 13 13 13 13 10 12 12
04/30 Read block 594 len 24576
05/30 Read block 242 len 32768
06/30 Write block 241 len 40960 data: 6 7 9 10 9 11 13 13 11 8
07/30 Write block 445 len 8192 data: 7 7
08/30 Read block 123 len 40960
09/30 Read block 407 len 28672
10/30 Write block 98 len 28672 data: 9 9 11 12 11 11 14
11/30 Read block 336 len 40960
12/30 Write block 493 len 12288 data: 12 9 14
13/30 Flush
14/30 Write block 416 len 28672 data: 7 9 10 11 11 9 9
15/30 Write block 115 len 28672 data: 9 12 13 11 14 11 11
16/30 Read block 65 len 40960
17/30 Read block 97 len 16384
18/30 Read block 551 len 24576
19/30 Flush
20/30 Write block 405 len 40960 data: 10 9 8 7 8 7 7 6 6 6
21/30 Read block 52 len 16384
22/30 Read block 252 len 12288
23/30 Write block 460 len 32768 data: 9 8 11 10 9 8 7 7
24/30 Flush
25/30 Write block 529 len 32768 data: 14 14 13 13 12 11 13 10
26/30 Write block 180 len 24576 data: 13 17 15 13 9 7
27/30 Write block 260 len 24576 data: 9 10 9 9 10 11
28/30 Write block 103 len 16384 data: 12 15 15 13
29/30 Write block 153 len 28672 data: 17 15 9 10 9 9 8
30/30 Write block 340 len 24576 data: 8 9 11 11 13 11
----------------------------------------------------------------
Crucible gen:114 GIO:true work queues: Upstairs:0 downstairs:6
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1024 Acked Write 8 Done Done Done false
1025 Acked Write 6 Done Done Done false
1026 Acked Write 6 Done Done Done false
1027 Acked Write 4 Done Done Done false
1028 Acked Write 7 Sent Sent Sent false
1029 Acked Write 6 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 4 4 4 12
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1023 1023 1023
Downstairs last five completed: 1023 1022 1021 1020 1019
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:02.33504441Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1638}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 113 113 113 206 206 206 F F F
1 020-039 112 112 112 204 204 204 F F F
2 040-059 106 106 106 194 194 194 F F F
3 060-079 114 114 114 207 207 207 F F F
4 080-099 114 114 114 207 207 207 F F F
5 100-119 114 114 114 208 208 208 T T T
6 120-139 114 114 114 208 208 208 F F F
7 140-159 113 113 113 206 206 206 T T T
8 160-179 110 110 110 199 199 199 F F F
9 180-199 110 110 110 201 201 201 T T T
10 200-219 108 108 108 197 197 197 F F F
11 220-239 112 112 112 204 204 204 F F F
12 240-259 114 114 114 207 207 207 F F F
13 260-279 111 111 111 202 202 202 T T T
14 280-299 112 112 112 205 205 205 F F F
15 300-319 104 104 104 192 192 192 F F F
16 320-339 114 114 114 207 207 207 F F F
17 340-359 114 114 114 207 207 207 T T T
18 360-379 113 113 113 206 206 206 F F F
19 380-399 108 108 108 197 197 197 F F F
20 400-419 114 114 114 209 209 209 F F F
21 420-439 114 114 114 208 208 208 F F F
22 440-459 114 114 114 207 207 207 F F F
23 460-479 114 114 114 209 209 209 F F F
24 480-499 114 114 114 207 207 207 F F F
25 500-519 111 111 111 202 202 202 F F F
26 520-539 111 111 111 202 202 202 T T T
27 540-559 112 112 112 205 205 205 F F F
28 560-579 112 112 112 205 205 205 F F F
29 580-599 112 112 112 203 203 203 F F F
Max gen: 114, Max flush: 209
A Difference in extent metadata is expected here
On loop 57
Verifying data now
No change after verify
Loop: 57 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 113 113 113 206 206 206 F F F
1 020-039 112 112 112 204 204 204 F F F
2 040-059 106 106 106 194 194 194 F F F
3 060-079 114 114 114 207 207 207 F F F
4 080-099 114 114 114 207 207 207 F F F
5 100-119 115 115 115 210 210 210 F F F
6 120-139 114 114 114 208 208 208 F F F
7 140-159 115 115 115 210 210 210 F F F
8 160-179 110 110 110 199 199 199 F F F
9 180-199 115 115 115 210 210 210 F F F
10 200-219 108 108 108 197 197 197 F F F
11 220-239 112 112 112 204 204 204 F F F
12 240-259 114 114 114 207 207 207 F F F
13 260-279 115 115 115 210 210 210 F F F
14 280-299 112 112 112 205 205 205 F F F
15 300-319 104 104 104 192 192 192 F F F
16 320-339 114 114 114 207 207 207 F F F
17 340-359 115 115 115 210 210 210 F F F
18 360-379 113 113 113 206 206 206 F F F
19 380-399 108 108 108 197 197 197 F F F
20 400-419 114 114 114 209 209 209 F F F
21 420-439 114 114 114 208 208 208 F F F
22 440-459 114 114 114 207 207 207 F F F
23 460-479 114 114 114 209 209 209 F F F
24 480-499 114 114 114 207 207 207 F F F
25 500-519 111 111 111 202 202 202 F F F
26 520-539 115 115 115 210 210 210 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 112 112 112 205 205 205 F F F
29 580-599 112 112 112 203 203 203 F F F
Max gen: 115, Max flush: 210
Begin loop 58
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.603165184Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.649838282Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.650865166Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.650919352Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.65093842Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.650960777Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646}
{"msg":"Crucible 464ba566-baa3-4db5-ae0b-79da3f9aeb36 has session id: 95f18b3f-b857-46ab-ba6f-09e33a607e63","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.651483303Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"Upstairs opts: Upstairs UUID: 464ba566-baa3-4db5-ae0b-79da3f9aeb36, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.651615212Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"Crucible stats registered with UUID: 464ba566-baa3-4db5-ae0b-79da3f9aeb36","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.651647871Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:06.651669218Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.652092716Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"task":"crutest"}
{"msg":"The guest has requested activation with gen:116","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.652137228Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646}
{"msg":"464ba566-baa3-4db5-ae0b-79da3f9aeb36 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.652161353Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.652175175Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"0","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.652200749Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"0","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.65221506Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"1","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.65223127Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"1","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.652254426Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"2","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.652281869Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"2","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.652308272Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"io task","client":"2","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.652334066Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"io task","client":"0","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.65236026Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"io task","client":"1","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50082)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.65255476Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"io task","client":"0","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"ds_connection connected from Ok(127.0.0.1:35227)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.652726933Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"io task","client":"2","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52893)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.652759253Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"io task","client":"1","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.653355694Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"0","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.653423192Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"2","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.653968504Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"2","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.654133722Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"0","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.654554132Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.654736588Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.654939643Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"1","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.655473562Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"1","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"[0]R flush_numbers[0..12]: [206, 204, 194, 207, 207, 210, 208, 210, 199, 210, 197, 204]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.655986095Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"downstairs","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"[0]R generation[0..12]: [113, 112, 106, 114, 114, 115, 114, 115, 110, 115, 108, 112]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656009191Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"downstairs","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656025141Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"downstairs","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"[1]R flush_numbers[0..12]: [206, 204, 194, 207, 207, 210, 208, 210, 199, 210, 197, 204]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656039052Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"downstairs","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"[1]R generation[0..12]: [113, 112, 106, 114, 114, 115, 114, 115, 110, 115, 108, 112]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656052893Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"downstairs","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656067165Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"downstairs","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"[2]R flush_numbers[0..12]: [206, 204, 194, 207, 207, 210, 208, 210, 199, 210, 197, 204]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656080966Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"downstairs","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"[2]R generation[0..12]: [113, 112, 106, 114, 114, 115, 114, 115, 110, 115, 108, 112]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656099335Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"downstairs","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656113536Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"downstairs","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"Max found gen is 116","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656126798Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"downstairs","":"downstairs","Wait for a query_work_queue command to finish before sending IO
session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"Generation requested: 116 >= found:116","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656147345Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"downstairs","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"Next flush: 211","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656160457Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"downstairs","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656173329Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"":"downstairs","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656186581Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656198583Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"0","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656211745Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"1","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656224577Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"client":"2","":"downstairs","session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656237639Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656249682Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"464ba566-baa3-4db5-ae0b-79da3f9aeb36 is now active with session: 95f18b3f-b857-46ab-ba6f-09e33a607e63","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656262374Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656274246Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646,"session_id":"95f18b3f-b857-46ab-ba6f-09e33a607e63"}
{"msg":"The guest has finished waiting for activation with:116","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:06.656286249Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 113 len 20480 data: 9 7 10 13 14
02/30 Flush
03/30 Write block 224 len 40960 data: 8 6 7 7 7 6 8 8 5 5
04/30 Write block 102 len 24576 data: 12 13 16 16 14 9
05/30 Write block 426 len 28672 data: 8 8 7 9 11 10 13
06/30 Write block 399 len 16384 data: 5 8 9 10
07/30 Write block 320 len 12288 data: 6 7 8
08/30 Read block 60 len 28672
09/30 Write block 435 len 36864 data: 14 10 6 6 5 7 7 7 6
10/30 Flush
11/30 Write block 436 len 4096 data: 11
12/30 Read block 56 len 16384
13/30 Write block 271 len 40960 data: 11 10 10 13 15 13 14 15 13 12
14/30 Write block 310 len 28672 data: 10 9 9 8 11 10 7
15/30 Read block 377 len 32768
16/30 Read block 344 len 24576
17/30 Write block 58 len 8192 data: 8 7
18/30 Write block 292 len 24576 data: 11 11 10 10 12 11
19/30 Write block 182 len 24576 data: 16 14 10 8 7 8
20/30 Read block 278 len 40960
21/30 Read block 188 len 24576
22/30 Read block 71 len 12288
23/30 Write block 312 len 20480 data: 10 9 12 11 8
24/30 Write block 71 len 4096 data: 13
25/30 Write block 185 len 36864 data: 9 8 9 5 5 5 9 8 9
26/30 Read block 268 len 36864
27/30 Write block 242 len 24576 data: 8 10 11 10 12 14
28/30 Read block 334 len 28672
29/30 Write block 470 len 8192 data: 7 8
30/30 Read block 214 len 8192
----------------------------------------------------------------
Crucible gen:116 GIO:true work queues: Upstairs:0 downstairs:20
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1010 Acked Write 1 Done Done Done false
1011 Acked Read 4 Done Done Done false
1012 Acked Write 10 Done Done Done false
1013 Acked Write 7 Done Done Done false
1014 Acked Read 8 Done Done Done false
1015 Acked Read 6 Done Done Done false
1016 Acked Write 2 Done Done Done false
1017 Acked Write 6 Done Done Done false
1018 Acked Write 6 Done Done Done false
1019 Acked Read 10 Done Done Done false
1020 Acked Read 6 Done Done Done false
1021 Acked Read 3 Done Done Done false
1022 Acked Write 5 Done Done Done false
1023 Acked Write 1 Done Done Done false
1024 Acked Write 9 Done Done Done false
1025 Acked Read 9 Done Done Done false
1026 Acked Write 6 Done Done Done false
1027 Acked Read 7 Done Done Done false
1028 Acked Write 2 Done Done Done false
1029 Acked Read 2 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 20 20 20 60
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1009 1009 1009
Downstairs last five completed: 1009 1008 1007 1006 1005
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:06.676593014Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1646}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 113 113 113 206 206 206 F F F
1 020-039 112 112 112 204 204 204 F F F
2 040-059 106 106 106 194 194 194 T T T
3 060-079 114 114 114 207 207 207 T T T
4 080-099 114 114 114 207 207 207 F F F
5 100-119 116 116 116 212 212 212 F F F
6 120-139 114 114 114 208 208 208 F F F
7 140-159 115 115 115 210 210 210 F F F
8 160-179 110 110 110 199 199 199 F F F
9 180-199 115 115 115 210 210 210 T T T
10 200-219 108 108 108 197 197 197 F F F
11 220-239 116 116 116 212 212 212 F F F
12 240-259 114 114 114 207 207 207 T T T
13 260-279 115 115 115 210 210 210 T T T
14 280-299 112 112 112 205 205 205 T T T
15 300-319 104 104 104 192 192 192 T T T
16 320-339 116 116 116 212 212 212 F F F
17 340-359 115 115 115 210 210 210 F F F
18 360-379 113 113 113 206 206 206 F F F
19 380-399 116 116 116 212 212 212 F F F
20 400-419 116 116 116 212 212 212 F F F
21 420-439 116 116 116 212 212 212 T T T
22 440-459 116 116 116 212 212 212 F F F
23 460-479 114 114 114 209 209 209 T T T
24 480-499 114 114 114 207 207 207 F F F
25 500-519 111 111 111 202 202 202 F F F
26 520-539 115 115 115 210 210 210 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 112 112 112 205 205 205 F F F
29 580-599 112 112 112 203 203 203 F F F
Max gen: 116, Max flush: 212
A Difference in extent metadata is expected here
On loop 58
Verifying data now
No change after verify
Loop: 58 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 113 113 113 206 206 206 F F F
1 020-039 112 112 112 204 204 204 F F F
2 040-059 117 117 117 213 213 213 F F F
3 060-079 117 117 117 213 213 213 F F F
4 080-099 114 114 114 207 207 207 F F F
5 100-119 116 116 116 212 212 212 F F F
6 120-139 114 114 114 208 208 208 F F F
7 140-159 115 115 115 210 210 210 F F F
8 160-179 110 110 110 199 199 199 F F F
9 180-199 117 117 117 213 213 213 F F F
10 200-219 108 108 108 197 197 197 F F F
11 220-239 116 116 116 212 212 212 F F F
12 240-259 117 117 117 213 213 213 F F F
13 260-279 117 117 117 213 213 213 F F F
14 280-299 117 117 117 213 213 213 F F F
15 300-319 117 117 117 213 213 213 F F F
16 320-339 116 116 116 212 212 212 F F F
17 340-359 115 115 115 210 210 210 F F F
18 360-379 113 113 113 206 206 206 F F F
19 380-399 116 116 116 212 212 212 F F F
20 400-419 116 116 116 212 212 212 F F F
21 420-439 117 117 117 213 213 213 F F F
22 440-459 116 116 116 212 212 212 F F F
23 460-479 117 117 117 213 213 213 F F F
24 480-499 114 114 114 207 207 207 F F F
25 500-519 111 111 111 202 202 202 F F F
26 520-539 115 115 115 210 210 210 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 112 112 112 205 205 205 F F F
29 580-599 112 112 112 203 203 203 F F F
Max gen: 117, Max flush: 213
Begin loop 59
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:10.954843216Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.000240484Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.001333087Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.00139281Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.001419923Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.001438292Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654}
{"msg":"Crucible 45e21778-9520-44ed-856e-d43b9c784bc9 has session id: fff60fe3-d220-4ce3-a443-b32df50d5e61","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.002260763Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"Upstairs opts: Upstairs UUID: 45e21778-9520-44ed-856e-d43b9c784bc9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.002318317Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"Crucible stats registered with UUID: 45e21778-9520-44ed-856e-d43b9c784bc9","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.002343871Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:11.002359552Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.002820796Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"task":"crutest"}
{"msg":"The guest has requested activation with gen:118","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.002853866Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654}
{"msg":"45e21778-9520-44ed-856e-d43b9c784bc9 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.002946608Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.002969314Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"0","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.002999405Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"0","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.003022231Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"1","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.003051303Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"1","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.003084373Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"2","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.00312092Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"2","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.003160716Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"io task","client":"2","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.003190537Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"io task","client":"0","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.003220439Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"io task","client":"1","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"ds_connection connected from Ok(127.0.0.1:48218)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.003568643Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"io task","client":"1","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"ds_connection connected from Ok(127.0.0.1:55169)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.003711175Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"io task","client":"2","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"ds_connection connected from Ok(127.0.0.1:34002)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.003746113Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"io task","client":"0","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.004587032Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"0","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.004626338Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"1","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.005360783Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"2","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.005546798Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"1","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.005586783Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"0","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006009871Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"2","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.00630198Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006337948Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"[0]R flush_numbers[0..12]: [206, 204, 213, 213, 207, 212, 208, 210, 199, 213, 197, 212]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.00670582Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"downstairs","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"[0]R generation[0..12]: [113, 112, 117, 117, 114, 116, 114, 115, 110, 117, 108, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006735902Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"downstairs","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006752631Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"downstairs","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"[1]R flush_numbers[0..12]: [206, 204, 213, 213, 207, 212, 208, 210, 199, 213, 197, 212]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006766732Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"downstairs","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"[1]R generation[0..12]: [113, 112, 117, 117, 114, 116, 114, 115, 110, 117, 108, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006780364Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"downstairs","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006794555Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"downstairs","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"[2]R flush_numbers[0..12]: [206, 204, 213, 213, 207, 212, 208, 210, 199, 213, 197, 212]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006808137Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"downstairs","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"[2]R generation[0..12]: [113, 112, 117, 117, 114, 116, 114, 115, 110, 117, 108, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006821649Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"downstairs","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.00683534Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"downstairs","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"Max found gen is 118","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006848172Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"downstairs","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
Wait for a query_work_queue command to finish before sending IO
{"msg":"Generation requested: 118 >= found:118","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.00686771Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"downstairs","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"Next flush: 214","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006880912Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"downstairs","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006893724Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"":"downstairs","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006906466Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006923066Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"0","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006936637Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"1","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006949219Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"client":"2","":"downstairs","session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006961982Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006973844Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"45e21778-9520-44ed-856e-d43b9c784bc9 is now active with session: fff60fe3-d220-4ce3-a443-b32df50d5e61","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006986467Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.006998159Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654,"session_id":"fff60fe3-d220-4ce3-a443-b32df50d5e61"}
{"msg":"The guest has finished waiting for activation with:118","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:11.007010402Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 300 len 16384 data: 9 11 9 11
02/30 Flush
03/30 Flush
04/30 Write block 242 len 20480 data: 9 11 12 11 13
05/30 Write block 402 len 28672 data: 11 10 10 11 10 9 8
06/30 Flush
07/30 Write block 300 len 40960 data: 10 12 10 12 10 11 12 12 12 11
08/30 Write block 374 len 36864 data: 14 15 16 15 15 14 10 8 5
09/30 Read block 498 len 20480
10/30 Flush
11/30 Write block 280 len 24576 data: 13 12 10 6 5 5
12/30 Flush
13/30 Write block 52 len 32768 data: 8 7 8 9 8 8 9 8
14/30 Flush
15/30 Flush
16/30 Write block 422 len 24576 data: 10 11 9 9 9 9
17/30 Read block 523 len 36864
18/30 Write block 568 len 32768 data: 7 6 8 7 8 9 8 9
19/30 Write block 483 len 12288 data: 10 11 10
20/30 Write block 190 len 16384 data: 6 10 9 10
21/30 Read block 73 len 16384
22/30 Write block 459 len 40960 data: 8 10 9 12 11 10 9 8 8 7
23/30 Flush
24/30 Write block 496 len 12288 data: 14 12 12
25/30 Read block 574 len 4096
26/30 Write block 24 len 16384 data: 14 12 12 10
27/30 Read block 131 len 4096
28/30 Write block 368 len 16384 data: 8 10 11 12
29/30 Write block 18 len 32768 data: 5 6 6 8 11 13 15 13
30/30 Write block 386 len 24576 data: 7 7 8 8 10 10
----------------------------------------------------------------
Crucible gen:118 GIO:true work queues: Upstairs:0 downstairs:7
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1023 Acked Write 3 Done Done Done false
1024 Acked Read 1 Done Done Done false
1025 Acked Write 4 Done Done Done false
1026 Acked Read 1 Done Done Done false
1027 Acked Write 4 Done Done Done false
1028 Acked Write 8 Sent Sent Sent false
1029 Acked Write 6 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 5 5 5 15
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1022 1022 1022
Downstairs last five completed: 1022 1021 1020 1019 1018
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:11.027550612Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1654}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 113 113 113 206 206 206 T T T
1 020-039 112 112 112 204 204 204 T T T
2 040-059 118 118 118 219 219 219 F F F
3 060-079 117 117 117 213 213 213 F F F
4 080-099 114 114 114 207 207 207 F F F
5 100-119 116 116 116 212 212 212 F F F
6 120-139 114 114 114 208 208 208 F F F
7 140-159 115 115 115 210 210 210 F F F
8 160-179 110 110 110 199 199 199 F F F
9 180-199 118 118 118 221 221 221 F F F
10 200-219 108 108 108 197 197 197 F F F
11 220-239 116 116 116 212 212 212 F F F
12 240-259 118 118 118 216 216 216 F F F
13 260-279 117 117 117 213 213 213 F F F
14 280-299 118 118 118 218 218 218 F F F
15 300-319 118 118 118 217 217 217 F F F
16 320-339 116 116 116 212 212 212 F F F
17 340-359 115 115 115 210 210 210 F F F
18 360-379 118 118 118 217 217 217 T T T
19 380-399 118 118 118 217 217 217 T T T
20 400-419 118 118 118 216 216 216 F F F
21 420-439 118 118 118 221 221 221 F F F
22 440-459 118 118 118 221 221 221 F F F
23 460-479 118 118 118 221 221 221 F F F
24 480-499 118 118 118 221 221 221 T T T
25 500-519 111 111 111 202 202 202 F F F
26 520-539 115 115 115 210 210 210 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 118 118 118 221 221 221 F F F
29 580-599 112 112 112 203 203 203 F F F
Max gen: 118, Max flush: 221
A Difference in extent metadata is expected here
On loop 59
Verifying data now
No change after verify
Loop: 59 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 119 119 119 222 222 222 F F F
1 020-039 119 119 119 222 222 222 F F F
2 040-059 118 118 118 219 219 219 F F F
3 060-079 117 117 117 213 213 213 F F F
4 080-099 114 114 114 207 207 207 F F F
5 100-119 116 116 116 212 212 212 F F F
6 120-139 114 114 114 208 208 208 F F F
7 140-159 115 115 115 210 210 210 F F F
8 160-179 110 110 110 199 199 199 F F F
9 180-199 118 118 118 221 221 221 F F F
10 200-219 108 108 108 197 197 197 F F F
11 220-239 116 116 116 212 212 212 F F F
12 240-259 118 118 118 216 216 216 F F F
13 260-279 117 117 117 213 213 213 F F F
14 280-299 118 118 118 218 218 218 F F F
15 300-319 118 118 118 217 217 217 F F F
16 320-339 116 116 116 212 212 212 F F F
17 340-359 115 115 115 210 210 210 F F F
18 360-379 119 119 119 222 222 222 F F F
19 380-399 119 119 119 222 222 222 F F F
20 400-419 118 118 118 216 216 216 F F F
21 420-439 118 118 118 221 221 221 F F F
22 440-459 118 118 118 221 221 221 F F F
23 460-479 118 118 118 221 221 221 F F F
24 480-499 119 119 119 222 222 222 F F F
25 500-519 111 111 111 202 202 202 F F F
26 520-539 115 115 115 210 210 210 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 118 118 118 221 221 221 F F F
29 580-599 112 112 112 203 203 203 F F F
Max gen: 119, Max flush: 222
Begin loop 60
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.28231015Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.327298621Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.32836549Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.328410752Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.328426022Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.328440883Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662}
{"msg":"Crucible ad19945d-0bcc-4b9b-8849-c0c32ca99af2 has session id: a28191f4-86e8-4859-9855-7a3c6fcb2e8a","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329068645Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"Upstairs opts: Upstairs UUID: ad19945d-0bcc-4b9b-8849-c0c32ca99af2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329105592Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"Crucible stats registered with UUID: ad19945d-0bcc-4b9b-8849-c0c32ca99af2","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329122202Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:15.329135273Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329644088Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"task":"crutest"}
{"msg":"The guest has requested activation with gen:120","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329693668Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662}
{"msg":"ad19945d-0bcc-4b9b-8849-c0c32ca99af2 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329832342Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329860045Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"0","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329891375Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"0","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329905766Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"1","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329919828Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"1","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329933559Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"2","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329947231Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"2","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329959713Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"io task","client":"2","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.329973044Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"io task","client":"0","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.32999567Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"io task","client":"1","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52478)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.330145108Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"io task","client":"0","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"ds_connection connected from Ok(127.0.0.1:58948)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.330193908Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"io task","client":"2","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"ds_connection connected from Ok(127.0.0.1:42818)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.330290458Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"io task","client":"1","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.330971186Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"2","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.331016558Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"0","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.331034727Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"1","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.331618436Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"0","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.331646928Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"2","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.331662149Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"1","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332267763Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332295856Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"[0]R flush_numbers[0..12]: [222, 222, 219, 213, 207, 212, 208, 210, 199, 221, 197, 212]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332365333Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"downstairs","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"[0]R generation[0..12]: [119, 119, 118, 117, 114, 116, 114, 115, 110, 118, 108, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332390148Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"downstairs","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332404419Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"downstairs","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"[1]R flush_numbers[0..12]: [222, 222, 219, 213, 207, 212, 208, 210, 199, 221, 197, 212]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.33241797Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"downstairs","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"[1]R generation[0..12]: [119, 119, 118, 117, 114, 116, 114, 115, 110, 118, 108, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332431522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"downstairs","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332445473Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"downstairs","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"[2]R flush_numbers[0..12]: [222, 222, 219, 213, 207, 212, 208, 210, 199, 221, 197, 212]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332459005Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"downstairs","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"[2]R generation[0..12]: [119, 119, 118, 117, 114, 116, 114, 115, 110, 118, 108, 116]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332472377Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"downstairs","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332485729Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"downstairs","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"Max found gen is 120","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332501989Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"downstairs","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"Generation requested: 120 >= found:120","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.33251511Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"downstairs","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"Next flush: 223","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:41:15.332527683Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"downstairs","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332547141Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"":"downstairs","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332559993Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332571695Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"0","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332584967Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"1","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332597669Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"client":"2","":"downstairs","session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332610451Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332622314Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"ad19945d-0bcc-4b9b-8849-c0c32ca99af2 is now active with session: a28191f4-86e8-4859-9855-7a3c6fcb2e8a","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332634556Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332646129Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662,"session_id":"a28191f4-86e8-4859-9855-7a3c6fcb2e8a"}
{"msg":"The guest has finished waiting for activation with:120","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:15.332658212Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 435 len 8192 data: 15 12
02/30 Write block 462 len 16384 data: 13 12 11 10
03/30 Write block 94 len 40960 data: 12 11 9 8 10 10 12 13 13 14
04/30 Read block 364 len 32768
05/30 Read block 414 len 28672
06/30 Read block 422 len 8192
07/30 Read block 297 len 32768
08/30 Flush
09/30 Write block 117 len 12288 data: 15 12 15
10/30 Write block 97 len 32768 data: 9 11 11 13 14 14 15 17
11/30 Read block 195 len 36864
12/30 Write block 439 len 20480 data: 6 8 8 8 7
13/30 Flush
14/30 Write block 195 len 32768 data: 12 10 11 11 8 7 6 8
15/30 Write block 306 len 16384 data: 13 13 13 12
16/30 Write block 246 len 40960 data: 14 15 14 12 9 7 7 9 11 12
17/30 Write block 230 len 4096 data: 9
18/30 Write block 338 len 40960 data: 8 9 9 10 12 12 14 12 10 9
19/30 Read block 278 len 16384
20/30 Read block 155 len 40960
21/30 Write block 393 len 40960 data: 11 9 9 8 7 6 6 9 10 12
22/30 Flush
23/30 Write block 291 len 12288 data: 10 12 12
24/30 Read block 117 len 16384
25/30 Write block 389 len 24576 data: 9 11 11 12 12 10
26/30 Write block 334 len 20480 data: 4 5 6 7 9
27/30 Write block 229 len 28672 data: 7 10 9 6 6 6 7
28/30 Read block 135 len 36864
29/30 Write block 126 len 12288 data: 5 5 5
30/30 Read block 107 len 24576
----------------------------------------------------------------
Crucible gen:120 GIO:true work queues: Upstairs:0 downstairs:8
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1022 Acked Write 3 Done Done Done false
1023 Acked Read 4 Done Done Done false
1024 Acked Write 6 Done Done Done false
1025 Acked Write 5 Done Done Done false
1026 Acked Write 7 Done Done Done false
1027 Acked Read 9 Done Done Done false
1028 Acked Write 3 Done Done Done false
1029 Acked Read 6 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 0 1
Done 8 7 8 23
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1021 1021 1021
Downstairs last five completed: 1021 1020 1019 1018 1017
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:15.353507821Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1662}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 119 119 119 222 222 222 F F F
1 020-039 119 119 119 222 222 222 F F F
2 040-059 118 118 118 219 219 219 F F F
3 060-079 117 117 117 213 213 213 F F F
4 080-099 120 120 120 224 224 224 F F F
5 100-119 120 120 120 224 224 224 F F F
6 120-139 114 114 114 208 208 208 T T T
7 140-159 115 115 115 210 210 210 F F F
8 160-179 110 110 110 199 199 199 F F F
9 180-199 120 120 120 225 225 225 F F F
10 200-219 120 120 120 225 225 225 F F F
11 220-239 120 120 120 225 225 225 T T T
12 240-259 120 120 120 225 225 225 F F F
13 260-279 117 117 117 213 213 213 F F F
14 280-299 118 118 118 218 218 218 T T T
15 300-319 120 120 120 225 225 225 F F F
16 320-339 120 120 120 225 225 225 T T T
17 340-359 120 120 120 225 225 225 F F F
18 360-379 119 119 119 222 222 222 F F F
19 380-399 120 120 120 225 225 225 T T T
20 400-419 120 120 120 225 225 225 F F F
21 420-439 120 120 120 224 224 224 F F F
22 440-459 120 120 120 224 224 224 F F F
23 460-479 120 120 120 223 223 223 F F F
24 480-499 119 119 119 222 222 222 F F F
25 500-519 111 111 111 202 202 202 F F F
26 520-539 115 115 115 210 210 210 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 118 118 118 221 221 221 F F F
29 580-599 112 112 112 203 203 203 F F F
Max gen: 120, Max flush: 225
A Difference in extent metadata is expected here
On loop 60
Verifying data now
No change after verify
Loop: 60 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 119 119 119 222 222 222 F F F
1 020-039 119 119 119 222 222 222 F F F
2 040-059 118 118 118 219 219 219 F F F
3 060-079 117 117 117 213 213 213 F F F
4 080-099 120 120 120 224 224 224 F F F
5 100-119 120 120 120 224 224 224 F F F
6 120-139 121 121 121 226 226 226 F F F
7 140-159 115 115 115 210 210 210 F F F
8 160-179 110 110 110 199 199 199 F F F
9 180-199 120 120 120 225 225 225 F F F
10 200-219 120 120 120 225 225 225 F F F
11 220-239 121 121 121 226 226 226 F F F
12 240-259 120 120 120 225 225 225 F F F
13 260-279 117 117 117 213 213 213 F F F
14 280-299 121 121 121 226 226 226 F F F
15 300-319 120 120 120 225 225 225 F F F
16 320-339 121 121 121 226 226 226 F F F
17 340-359 120 120 120 225 225 225 F F F
18 360-379 119 119 119 222 222 222 F F F
19 380-399 121 121 121 226 226 226 F F F
20 400-419 120 120 120 225 225 225 F F F
21 420-439 120 120 120 224 224 224 F F F
22 440-459 120 120 120 224 224 224 F F F
23 460-479 120 120 120 223 223 223 F F F
24 480-499 119 119 119 222 222 222 F F F
25 500-519 111 111 111 202 202 202 F F F
26 520-539 115 115 115 210 210 210 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 118 118 118 221 221 221 F F F
29 580-599 112 112 112 203 203 203 F F F
Max gen: 121, Max flush: 226
Begin loop 61
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.611874536Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:19.656106406Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.656765188Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.689481398Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.690647567Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.690685023Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.690698765Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.690712806Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670}
{"msg":"Crucible b119a075-0f72-456e-b7e1-e711da07bee1 has session id: fa8ecfb3-7ddd-43cd-8879-b38d9445045c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.691404028Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"Upstairs opts: Upstairs UUID: b119a075-0f72-456e-b7e1-e711da07bee1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.691442264Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"Crucible stats registered with UUID: b119a075-0f72-456e-b7e1-e711da07bee1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.691459613Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:19.691472765Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692036007Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"task":"crutest"}
{"msg":"The guest has requested activation with gen:122","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692060881Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670}
{"msg":"b119a075-0f72-456e-b7e1-e711da07bee1 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692139093Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692157102Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"0","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692181586Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"0","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692195328Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"1","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692209459Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"1","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692222961Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"2","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692237922Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"2","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692251143Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"io task","client":"2","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692265055Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"io task","client":"0","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692295306Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"io task","client":"1","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:46413)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692457406Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"io task","client":"2","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:40827)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692500549Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"io task","client":"1","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49496)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.692529871Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"io task","client":"0","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.693285143Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"1","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.693313685Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"2","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.693574434Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"0","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.694051099Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"2","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.694091394Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"1","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.694116928Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"0","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.694627872Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.694740372Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"[0]R flush_numbers[0..12]: [222, 222, 219, 213, 224, 224, 226, 210, 199, 225, 225, 226]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.694849824Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"downstairs","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"[0]R generation[0..12]: [119, 119, 118, 117, 120, 120, 121, 115, 110, 120, 120, 121]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.694877827Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"downstairs","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.694897205Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"downstairs","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"[1]R flush_numbers[0..12]: [222, 222, 219, 213, 224, 224, 226, 210, 199, 225, 225, 226]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.694915064Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"downstairs","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"[1]R generation[0..12]: [119, 119, 118, 117, 120, 120, 121, 115, 110, 120, 120, 121]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.694939709Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"downstairs","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.694966522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"downstairs","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"[2]R flush_numbers[0..12]: [222, 222, 219, 213, 224, 224, 226, 210, 199, 225, 225, 226]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.694992226Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"downstairs","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"[2]R generation[0..12]: [119, 119, 118, 117, 120, 120, 121, 115, 110, 120, 120, 121]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:41:19.695016851Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"downstairs","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.695049111Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"downstairs","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"Max found gen is 122","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.695074105Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"downstairs","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"Generation requested: 122 >= found:122","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.695089006Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"downstairs","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"Next flush: 227","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.695101878Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"downstairs","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.695123185Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"":"downstairs","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.6951473Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.695170156Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"0","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.695187495Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"1","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.695200667Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"client":"2","":"downstairs","session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.695225981Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.695248517Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"b119a075-0f72-456e-b7e1-e711da07bee1 is now active with session: fa8ecfb3-7ddd-43cd-8879-b38d9445045c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.695273202Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.69529201Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670,"session_id":"fa8ecfb3-7ddd-43cd-8879-b38d9445045c"}
{"msg":"The guest has finished waiting for activation with:122","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:19.695311538Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 518 len 24576 data: 7 8 8 11 12 10
02/30 Flush
03/30 Write block 585 len 36864 data: 7 9 8 9 10 8 6 6 7
04/30 Write block 86 len 16384 data: 15 12 11 9
05/30 Read block 199 len 8192
06/30 Read block 383 len 4096
07/30 Read block 360 len 32768
08/30 Write block 204 len 40960 data: 10 10 10 11 12 10 7 9 9 9
09/30 Read block 6 len 8192
10/30 Read block 420 len 8192
11/30 Flush
12/30 Flush
13/30 Write block 56 len 8192 data: 9 9
14/30 Read block 254 len 16384
15/30 Read block 217 len 20480
16/30 Write block 456 len 4096 data: 8
17/30 Read block 440 len 12288
18/30 Read block 255 len 4096
19/30 Write block 415 len 36864 data: 5 8 10 11 12 12 10 11 12
20/30 Write block 121 len 8192 data: 12 10
21/30 Write block 319 len 20480 data: 6 7 8 9 12
22/30 Write block 89 len 40960 data: 10 12 11 14 15 13 12 10 10 12
23/30 Read block 69 len 36864
24/30 Read block 150 len 8192
25/30 Write block 157 len 24576 data: 10 10 9 5 11 9
26/30 Read block 171 len 4096
27/30 Read block 197 len 4096
28/30 Read block 28 len 32768
29/30 Read block 132 len 20480
30/30 Write block 17 len 16384 data: 4 6 7 7
----------------------------------------------------------------
Crucible gen:122 GIO:true work queues: Upstairs:0 downstairs:18
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1012 Acked Write 2 Done Done Done false
1013 Acked Read 4 Done Done Done false
1014 Acked Read 5 Done Done Done false
1015 Acked Write 1 Done Done Done false
1016 Acked Read 3 Done Done Done false
1017 Acked Read 1 Done Done Done false
1018 Acked Write 9 Done Done Done false
1019 Acked Write 2 Done Done Done false
1020 Acked Write 5 Done Done Done false
1021 Acked Write 10 Done Done Done false
1022 Acked Read 9 Done Done Done false
1023 Acked Read 2 Done Done Done false
1024 Acked Write 6 Done Done Done false
1025 Acked Read 1 Done Done Done false
1026 Acked Read 1 Done Done Done false
1027 Acked Read 8 Done Done Done false
1028 Acked Read 5 Done Done Done false
1029 Acked Write 4 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 17 17 17 51
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1011 1011 1011
Downstairs last five completed: 1011 1010 1009 1008 1007
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:19.715130595Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1670}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 119 119 119 222 222 222 T T T
1 020-039 119 119 119 222 222 222 T T T
2 040-059 118 118 118 219 219 219 T T T
3 060-079 117 117 117 213 213 213 F F F
4 080-099 122 122 122 228 228 228 T T T
5 100-119 120 120 120 224 224 224 F F F
6 120-139 121 121 121 226 226 226 T T T
7 140-159 115 115 115 210 210 210 T T T
8 160-179 110 110 110 199 199 199 T T T
9 180-199 120 120 120 225 225 225 F F F
10 200-219 122 122 122 228 228 228 F F F
11 220-239 121 121 121 226 226 226 F F F
12 240-259 120 120 120 225 225 225 F F F
13 260-279 117 117 117 213 213 213 F F F
14 280-299 121 121 121 226 226 226 F F F
15 300-319 120 120 120 225 225 225 T T T
16 320-339 121 121 121 226 226 226 T T T
17 340-359 120 120 120 225 225 225 F F F
18 360-379 119 119 119 222 222 222 F F F
19 380-399 121 121 121 226 226 226 F F F
20 400-419 120 120 120 225 225 225 T T T
21 420-439 120 120 120 224 224 224 T T T
22 440-459 120 120 120 224 224 224 T T T
23 460-479 120 120 120 223 223 223 F F F
24 480-499 119 119 119 222 222 222 F F F
25 500-519 122 122 122 227 227 227 F F F
26 520-539 122 122 122 227 227 227 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 118 118 118 221 221 221 F F F
29 580-599 122 122 122 228 228 228 F F F
Max gen: 122, Max flush: 228
A Difference in extent metadata is expected here
On loop 61
Verifying data now
No change after verify
Loop: 61 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 123 123 123 229 229 229 F F F
1 020-039 123 123 123 229 229 229 F F F
2 040-059 123 123 123 229 229 229 F F F
3 060-079 117 117 117 213 213 213 F F F
4 080-099 123 123 123 229 229 229 F F F
5 100-119 120 120 120 224 224 224 F F F
6 120-139 123 123 123 229 229 229 F F F
7 140-159 123 123 123 229 229 229 F F F
8 160-179 123 123 123 229 229 229 F F F
9 180-199 120 120 120 225 225 225 F F F
10 200-219 122 122 122 228 228 228 F F F
11 220-239 121 121 121 226 226 226 F F F
12 240-259 120 120 120 225 225 225 F F F
13 260-279 117 117 117 213 213 213 F F F
14 280-299 121 121 121 226 226 226 F F F
15 300-319 123 123 123 229 229 229 F F F
16 320-339 123 123 123 229 229 229 F F F
17 340-359 120 120 120 225 225 225 F F F
18 360-379 119 119 119 222 222 222 F F F
19 380-399 121 121 121 226 226 226 F F F
20 400-419 123 123 123 229 229 229 F F F
21 420-439 123 123 123 229 229 229 F F F
22 440-459 123 123 123 229 229 229 F F F
23 460-479 120 120 120 223 223 223 F F F
24 480-499 119 119 119 222 222 222 F F F
25 500-519 122 122 122 227 227 227 F F F
26 520-539 122 122 122 227 227 227 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 118 118 118 221 221 221 F F F
29 580-599 122 122 122 228 228 228 F F F
Max gen: 123, Max flush: 229
Begin loop 62
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.047391652Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.093897353Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.094943785Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.09498427Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.09500097Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.095015261Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678}
{"msg":"Crucible 48567d15-775a-49c4-97e7-c2ed82c2ed6d has session id: 0338fde3-0391-4103-af4f-aeac50e36189","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.095615479Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"Upstairs opts: Upstairs UUID: 48567d15-775a-49c4-97e7-c2ed82c2ed6d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.095660341Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"Crucible stats registered with UUID: 48567d15-775a-49c4-97e7-c2ed82c2ed6d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.095680729Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:24.095708552Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096192792Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"task":"crutest"}
{"msg":"The guest has requested activation with gen:124","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096218166Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678}
{"msg":"48567d15-775a-49c4-97e7-c2ed82c2ed6d active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096306651Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096332525Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"0","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096364595Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"0","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096379456Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"1","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096393817Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"1","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096407029Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"2","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096426837Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"2","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096451172Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"io task","client":"2","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096476156Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"io task","client":"0","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096499771Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"io task","client":"1","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50246)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096524596Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"io task","client":"2","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"ds_connection connected from Ok(127.0.0.1:56782)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096552449Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"io task","client":"1","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"ds_connection connected from Ok(127.0.0.1:46541)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.096721694Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"io task","client":"0","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.097283416Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"1","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.097318315Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"0","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.097339142Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"2","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.09800416Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"2","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098032822Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"0","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098048782Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"1","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098613543Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098716459Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"[0]R flush_numbers[0..12]: [229, 229, 229, 213, 229, 224, 229, 229, 229, 225, 228, 226]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098759482Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"downstairs","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"[0]R generation[0..12]: [123, 123, 123, 117, 123, 120, 123, 123, 123, 120, 122, 121]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098776522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"downstairs","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098790823Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"downstairs","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"[1]R flush_numbers[0..12]: [229, 229, 229, 213, 229, 224, 229, 229, 229, 225, 228, 226]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098804415Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"downstairs","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"[1]R generation[0..12]: [123, 123, 123, 117, 123, 120, 123, 123, 123, 120, 122, 121]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098817946Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"downstairs","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098831607Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"downstairs","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"[2]R flush_numbers[0..12]: [229, 229, 229, 213, 229, 224, 229, 229, 229, 225, 228, 226]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098844869Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"downstairs","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"[2]R generation[0..12]: [123, 123, 123, 117, 123, 120, 123, 123, 123, 120, 122, 121]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098858051Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"downstairs","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098871483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"downstairs","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"Max found gen is 124","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098888112Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"downstairs","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"Generation requested: 124 >= found:124","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098901534Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"downstairs","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"Next flush: 230","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:41:24.098914306Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"downstairs","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098933055Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"":"downstairs","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098945997Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.09895782Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"0","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098971171Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"1","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.098983943Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"client":"2","":"downstairs","session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.099013115Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
,"time":"2025-01-27T19:41:24.099026307Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"48567d15-775a-49c4-97e7-c2ed82c2ed6d is now active with session: 0338fde3-0391-4103-af4f-aeac50e36189","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.099040178Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.099052091Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678,"session_id":"0338fde3-0391-4103-af4f-aeac50e36189"}
{"msg":"The guest has finished waiting for activation with:124","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:24.099064084Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 146 len 16384 data: 13 13 13 14
02/30 Write block 488 len 36864 data: 12 9 11 12 14 13 10 15 15
03/30 Write block 89 len 20480 data: 11 13 12 15 16
04/30 Write block 185 len 16384 data: 10 9 10 6
05/30 Write block 391 len 4096 data: 12
06/30 Write block 407 len 20480 data: 10 9 9 8 8
07/30 Read block 228 len 12288
08/30 Write block 346 len 16384 data: 11 10 12 13
09/30 Flush
10/30 Write block 381 len 36864 data: 9 6 5 5 5 8 8 9 10
11/30 Write block 395 len 28672 data: 10 9 8 7 7 10 11
12/30 Write block 518 len 32768 data: 8 9 9 12 13 11 11 10
13/30 Write block 591 len 24576 data: 7 7 8 6 6 8
14/30 Flush
15/30 Write block 241 len 12288 data: 7 10 12
16/30 Write block 268 len 40960 data: 7 10 10 12 11 11 14 16 14 15
17/30 Read block 30 len 24576
18/30 Write block 193 len 4096 data: 11
19/30 Read block 470 len 4096
20/30 Write block 265 len 20480 data: 12 10 7 8 11
21/30 Write block 230 len 4096 data: 11
22/30 Read block 357 len 20480
23/30 Read block 280 len 8192
24/30 Read block 8 len 20480
25/30 Read block 398 len 8192
26/30 Read block 197 len 24576
27/30 Write block 64 len 32768 data: 11 14 14 14 14 11 13 14
28/30 Read block 574 len 16384
29/30 Write block 491 len 12288 data: 13 15 14
30/30 Write block 293 len 8192 data: 13 11
----------------------------------------------------------------
Crucible gen:124 GIO:true work queues: Upstairs:0 downstairs:16
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1014 Acked Write 3 Done Done Done false
1015 Acked Write 10 Done Done Done false
1016 Acked Read 6 Done Done Done false
1017 Acked Write 1 Done Done Done false
1018 Acked Read 1 Done Done Done false
1019 Acked Write 5 Done Done Done false
1020 Acked Write 1 Done Done Done false
1021 Acked Read 5 Done Done Done false
1022 Acked Read 2 Done Done Done false
1023 Acked Read 5 Done Done Done false
1024 Acked Read 2 Done Done Done false
1025 Acked Read 6 Done Done Done false
1026 Acked Write 8 Done Done Done false
1027 Acked Read 4 Done Done Done false
1028 Acked Write 3 Sent Sent Sent false
1029 Acked Write 2 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 14 14 14 42
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1013 1013 1013
Downstairs last five completed: 1013 1012 1011 1010 1009
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:24.116804398Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1678}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 123 123 123 229 229 229 F F F
1 020-039 123 123 123 229 229 229 F F F
2 040-059 123 123 123 229 229 229 F F F
3 060-079 117 117 117 213 213 213 T T T
4 080-099 124 124 124 230 230 230 F F F
5 100-119 120 120 120 224 224 224 F F F
6 120-139 123 123 123 229 229 229 F F F
7 140-159 124 124 124 230 230 230 F F F
8 160-179 123 123 123 229 229 229 F F F
9 180-199 124 124 124 230 230 230 T T T
10 200-219 122 122 122 228 228 228 F F F
11 220-239 121 121 121 226 226 226 T T T
12 240-259 120 120 120 225 225 225 T T T
13 260-279 117 117 117 213 213 213 T T T
14 280-299 121 121 121 226 226 226 T T T
15 300-319 123 123 123 229 229 229 F F F
16 320-339 123 123 123 229 229 229 F F F
17 340-359 124 124 124 230 230 230 F F F
18 360-379 119 119 119 222 222 222 F F F
19 380-399 124 124 124 231 231 231 F F F
20 400-419 124 124 124 231 231 231 F F F
21 420-439 123 123 123 229 229 229 F F F
22 440-459 123 123 123 229 229 229 F F F
23 460-479 120 120 120 223 223 223 F F F
24 480-499 124 124 124 230 230 230 T T T
25 500-519 124 124 124 231 231 231 F F F
26 520-539 124 124 124 231 231 231 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 118 118 118 221 221 221 F F F
29 580-599 124 124 124 231 231 231 F F F
Max gen: 124, Max flush: 231
A Difference in extent metadata is expected here
On loop 62
Verifying data now
No change after verify
Loop: 62 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 123 123 123 229 229 229 F F F
1 020-039 123 123 123 229 229 229 F F F
2 040-059 123 123 123 229 229 229 F F F
3 060-079 125 125 125 232 232 232 F F F
4 080-099 124 124 124 230 230 230 F F F
5 100-119 120 120 120 224 224 224 F F F
6 120-139 123 123 123 229 229 229 F F F
7 140-159 124 124 124 230 230 230 F F F
8 160-179 123 123 123 229 229 229 F F F
9 180-199 125 125 125 232 232 232 F F F
10 200-219 122 122 122 228 228 228 F F F
11 220-239 125 125 125 232 232 232 F F F
12 240-259 125 125 125 232 232 232 F F F
13 260-279 125 125 125 232 232 232 F F F
14 280-299 125 125 125 232 232 232 F F F
15 300-319 123 123 123 229 229 229 F F F
16 320-339 123 123 123 229 229 229 F F F
17 340-359 124 124 124 230 230 230 F F F
18 360-379 119 119 119 222 222 222 F F F
19 380-399 124 124 124 231 231 231 F F F
20 400-419 124 124 124 231 231 231 F F F
21 420-439 123 123 123 229 229 229 F F F
22 440-459 123 123 123 229 229 229 F F F
23 460-479 120 120 120 223 223 223 F F F
24 480-499 125 125 125 232 232 232 F F F
25 500-519 124 124 124 231 231 231 F F F
26 520-539 124 124 124 231 231 231 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 118 118 118 221 221 221 F F F
29 580-599 124 124 124 231 231 231 F F F
Max gen: 125, Max flush: 232
Begin loop 63
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.38990258Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.437544218Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.438660507Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.438729364Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.438754769Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.438782871Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686}
{"msg":"Crucible 86128cb3-0f71-478c-9102-e36f5e45becd has session id: 660dc1e3-0fe5-4359-bc76-47ca92eca3b9","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.43935077Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"Upstairs opts: Upstairs UUID: 86128cb3-0f71-478c-9102-e36f5e45becd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.439412761Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"Crucible stats registered with UUID: 86128cb3-0f71-478c-9102-e36f5e45becd","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.439442843Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:28.439468537Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.439984287Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"task":"crutest"}
{"msg":"The guest has requested activation with gen:126","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440013559Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686}
{"msg":"86128cb3-0f71-478c-9102-e36f5e45becd active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440097597Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440120563Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"0","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440147536Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"0","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440161128Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"1","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440175109Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"1","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440188141Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"2","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440201953Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"2","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440215004Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"io task","client":"2","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440228396Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"io task","client":"0","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440250073Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"io task","client":"1","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"ds_connection connected from Ok(127.0.0.1:33918)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440356307Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"io task","client":"2","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47483)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440398431Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"io task","client":"1","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"ds_connection connected from Ok(127.0.0.1:38336)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.440427393Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"io task","client":"0","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.441151235Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"2","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.441182086Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"0","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.441283503Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"1","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.441807529Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"2","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.441838729Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"0","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.441909026Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"1","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442461214Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442490695Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"[0]R flush_numbers[0..12]: [229, 229, 229, 232, 230, 224, 229, 230, 229, 232, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442585707Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"downstairs","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"[0]R generation[0..12]: [123, 123, 123, 125, 124, 120, 123, 124, 123, 125, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442605874Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"downstairs","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442623523Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"downstairs","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"[1]R flush_numbers[0..12]: [229, 229, 229, 232, 230, 224, 229, 230, 229, 232, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442637764Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"downstairs","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"[1]R generation[0..12]: [123, 123, 123, 125, 124, 120, 123, 124, 123, 125, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442651776Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"downstairs","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442665587Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"downstairs","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"[2]R flush_numbers[0..12]: [229, 229, 229, 232, 230, 224, 229, 230, 229, 232, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442678989Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"downstairs","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"[2]R generation[0..12]: [123, 123, 123, 125, 124, 120, 123, 124, 123, 125, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.44269233Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"downstairs","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442707001Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"downstairs","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"Max found gen is 126","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442719544Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"downstairs","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"Generation requested: 126 >= found:126","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442732466Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"downstairs",Wait for a query_work_queue command to finish before sending IO
"":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"Next flush: 233","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442751724Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"downstairs","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442764436Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"":"downstairs","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442777378Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442789001Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"0","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442802013Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"1","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442814735Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"client":"2","":"downstairs","session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442830655Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442842877Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"86128cb3-0f71-478c-9102-e36f5e45becd is now active with session: 660dc1e3-0fe5-4359-bc76-47ca92eca3b9","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442855409Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442867192Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686,"session_id":"660dc1e3-0fe5-4359-bc76-47ca92eca3b9"}
{"msg":"The guest has finished waiting for activation with:126","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:28.442879225Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 349 len 32768 data: 14 13 12 10 9 9 8 9
02/30 Write block 322 len 4096 data: 10
03/30 Read block 80 len 20480
04/30 Write block 164 len 16384 data: 11 13 14 12
05/30 Write block 84 len 40960 data: 13 17 16 13 12 12 14 13 16 17
06/30 Write block 18 len 12288 data: 7 8 8
07/30 Write block 116 len 36864 data: 14 16 13 16 12 13 11 7 7
08/30 Read block 559 len 32768
09/30 Read block 236 len 24576
10/30 Flush
11/30 Write block 560 len 4096 data: 10
12/30 Read block 86 len 28672
13/30 Write block 82 len 32768 data: 12 11 14 18 17 14 13 13
14/30 Flush
15/30 Write block 34 len 36864 data: 8 8 5 5 6 7 9 10 8
16/30 Read block 487 len 16384
17/30 Read block 369 len 4096
18/30 Read block 455 len 40960
19/30 Read block 544 len 40960
20/30 Read block 141 len 40960
21/30 Read block 462 len 28672
22/30 Read block 488 len 8192
23/30 Write block 129 len 8192 data: 6 5
24/30 Write block 505 len 36864 data: 6 6 6 8 7 8 9 8 6
25/30 Write block 522 len 32768 data: 14 12 12 11 9 12 14 15
26/30 Write block 127 len 40960 data: 6 6 7 6 7 6 6 8 8 8
27/30 Write block 194 len 24576 data: 9 13 11 12 12 9
28/30 Read block 72 len 16384
29/30 Write block 587 len 40960 data: 9 10 11 9 8 8 9 7 7 9
30/30 Read block 351 len 32768
----------------------------------------------------------------
Crucible gen:126 GIO:true work queues: Upstairs:0 downstairs:16
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1014 Acked Write 9 Done Done Done false
1015 Acked Read 4 Done Done Done false
1016 Acked Read 1 Done Done Done false
1017 Acked Read 10 Done Done Done false
1018 Acked Read 10 Done Done Done false
1019 Acked Read 10 Done Done Done false
1020 Acked Read 7 Done Done Done false
1021 Acked Read 2 Done Done Done false
1022 Acked Write 2 Done Done Done false
1023 Acked Write 9 Done Done Done false
1024 Acked Write 8 Done Done Done false
1025 Acked Write 10 Done Done Done false
1026 Acked Write 6 Done Done Done false
1027 Acked Read 4 Done Done Done false
1028 Acked Write 10 Done Sent Done false
1029 Acked Read 8 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 2 1 3
Done 16 14 15 45
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1013 1013 1013
Downstairs last five completed: 1013 1012 1011 1010 1009
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:28.464280932Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1686}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 126 126 126 233 233 233 F F F
1 020-039 126 126 126 233 233 233 T T T
2 040-059 123 123 123 229 229 229 T T T
3 060-079 125 125 125 232 232 232 F F F
4 080-099 126 126 126 234 234 234 F F F
5 100-119 126 126 126 233 233 233 F F F
6 120-139 126 126 126 233 233 233 T T T
7 140-159 124 124 124 230 230 230 F F F
8 160-179 126 126 126 233 233 233 F F F
9 180-199 125 125 125 232 232 232 T T T
10 200-219 122 122 122 228 228 228 F F F
11 220-239 125 125 125 232 232 232 F F F
12 240-259 125 125 125 232 232 232 F F F
13 260-279 125 125 125 232 232 232 F F F
14 280-299 125 125 125 232 232 232 F F F
15 300-319 123 123 123 229 229 229 F F F
16 320-339 126 126 126 233 233 233 F F F
17 340-359 126 126 126 233 233 233 F F F
18 360-379 119 119 119 222 222 222 F F F
19 380-399 124 124 124 231 231 231 F F F
20 400-419 124 124 124 231 231 231 F F F
21 420-439 123 123 123 229 229 229 F F F
22 440-459 123 123 123 229 229 229 F F F
23 460-479 120 120 120 223 223 223 F F F
24 480-499 125 125 125 232 232 232 F F F
25 500-519 124 124 124 231 231 231 T T T
26 520-539 124 124 124 231 231 231 T T T
27 540-559 112 112 112 205 205 205 F F F
28 560-579 126 126 126 234 234 234 F F F
29 580-599 124 124 124 231 231 231 T T T
Max gen: 126, Max flush: 234
A Difference in extent metadata is expected here
On loop 63
Verifying data now
No change after verify
Loop: 63 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 126 126 126 233 233 233 F F F
1 020-039 127 127 127 235 235 235 F F F
2 040-059 127 127 127 235 235 235 F F F
3 060-079 125 125 125 232 232 232 F F F
4 080-099 126 126 126 234 234 234 F F F
5 100-119 126 126 126 233 233 233 F F F
6 120-139 127 127 127 235 235 235 F F F
7 140-159 124 124 124 230 230 230 F F F
8 160-179 126 126 126 233 233 233 F F F
9 180-199 127 127 127 235 235 235 F F F
10 200-219 122 122 122 228 228 228 F F F
11 220-239 125 125 125 232 232 232 F F F
12 240-259 125 125 125 232 232 232 F F F
13 260-279 125 125 125 232 232 232 F F F
14 280-299 125 125 125 232 232 232 F F F
15 300-319 123 123 123 229 229 229 F F F
16 320-339 126 126 126 233 233 233 F F F
17 340-359 126 126 126 233 233 233 F F F
18 360-379 119 119 119 222 222 222 F F F
19 380-399 124 124 124 231 231 231 F F F
20 400-419 124 124 124 231 231 231 F F F
21 420-439 123 123 123 229 229 229 F F F
22 440-459 123 123 123 229 229 229 F F F
23 460-479 120 120 120 223 223 223 F F F
24 480-499 125 125 125 232 232 232 F F F
25 500-519 127 127 127 235 235 235 F F F
26 520-539 127 127 127 235 235 235 F F F
27 540-559 112 112 112 205 205 205 F F F
28 560-579 126 126 126 234 234 234 F F F
29 580-599 127 127 127 235 235 235 F F F
Max gen: 127, Max flush: 235
Begin loop 64
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.738333933Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:32.781184028Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.781749578Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.813595027Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.814587972Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.81464012Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.814673679Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.814701142Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694}
{"msg":"Crucible b272162e-d46e-4ffd-bc14-a4efb0277f41 has session id: a9f6c17d-7bfc-40a7-9606-211876f4205f","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.815186022Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"Upstairs opts: Upstairs UUID: b272162e-d46e-4ffd-bc14-a4efb0277f41, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.815282662Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"Crucible stats registered with UUID: b272162e-d46e-4ffd-bc14-a4efb0277f41","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.815308666Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:32.815322708Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.815742228Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"task":"crutest"}
{"msg":"The guest has requested activation with gen:128","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.81577046Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694}
{"msg":"b272162e-d46e-4ffd-bc14-a4efb0277f41 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.815859675Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.815887658Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"0","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.81591652Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"0","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.81593134Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"1","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.815945962Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"1","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.815959003Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"2","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.815972885Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"2","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.815985817Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"io task","client":"2","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.815999189Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"io task","client":"0","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.81601257Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"io task","client":"1","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"ds_connection connected from Ok(127.0.0.1:60174)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.816102834Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"io task","client":"2","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"ds_connection connected from Ok(127.0.0.1:36589)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.816356537Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"io task","client":"1","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52182)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.816393554Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"io task","client":"0","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.816894734Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"1","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.816921967Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"2","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.817115747Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"0","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.817675551Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"1","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.817710099Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"2","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.817738561Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"0","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818307229Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818420559Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"[0]R flush_numbers[0..12]: [233, 235, 235, 232, 234, 233, 235, 230, 233, 235, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818459395Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"downstairs","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"[0]R generation[0..12]: [126, 127, 127, 125, 126, 126, 127, 124, 126, 127, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818474556Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"downstairs","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818488627Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"downstairs","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"[1]R flush_numbers[0..12]: [233, 235, 235, 232, 234, 233, 235, 230, 233, 235, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818502259Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"downstairs","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"[1]R generation[0..12]: [126, 127, 127, 125, 126, 126, 127, 124, 126, 127, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.81851579Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"downstairs","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818529552Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"downstairs","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"[2]R flush_numbers[0..12]: [233, 235, 235, 232, 234, 233, 235, 230, 233, 235, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818543443Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"downstairs","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"[2]R generation[0..12]: [126, 127, 127, 125, 126, 126, 127, 124, 126, 127, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818556805Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"downstairs","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818575094Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"downstairs","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"Max found gen is 128","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818588085Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"downstairs","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"Generation requested: 128 >= found:128","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818600818Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"downstairs","":"downstairs"Wait for a query_work_queue command to finish before sending IO
,"session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"Next flush: 236","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818622254Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"downstairs","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818634786Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"":"downstairs","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818647538Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818659171Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"0","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818672263Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"1","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818685015Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"client":"2","":"downstairs","session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818697997Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.81870985Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"b272162e-d46e-4ffd-bc14-a4efb0277f41 is now active with session: a9f6c17d-7bfc-40a7-9606-211876f4205f","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818722322Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818733835Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694,"session_id":"a9f6c17d-7bfc-40a7-9606-211876f4205f"}
{"msg":"The guest has finished waiting for activation with:128","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:32.818745728Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 317 len 16384 data: 6 5 7 8
02/30 Flush
03/30 Flush
04/30 Write block 107 len 20480 data: 10 10 10 10 10
05/30 Write block 571 len 40960 data: 8 9 10 9 10 8 8 6 6 7
06/30 Read block 487 len 16384
07/30 Read block 517 len 4096
08/30 Write block 548 len 8192 data: 10 13
09/30 Read block 53 len 16384
10/30 Read block 138 len 28672
11/30 Read block 339 len 36864
12/30 Write block 183 len 4096 data: 15
13/30 Write block 87 len 40960 data: 15 14 14 15 14 17 18 14 13 11
14/30 Write block 14 len 20480 data: 4 3 3 5 8
15/30 Write block 256 len 36864 data: 10 11 9 8 10 11 10 10 11
16/30 Write block 348 len 24576 data: 13 15 14 13 11 10
17/30 Write block 341 len 36864 data: 11 13 13 15 13 12 11 14 16
18/30 Read block 174 len 32768
19/30 Read block 534 len 24576
20/30 Read block 478 len 8192
21/30 Write block 434 len 4096 data: 13
22/30 Read block 483 len 8192
23/30 Read block 421 len 4096
24/30 Read block 259 len 8192
25/30 Write block 301 len 8192 data: 13 11
26/30 Read block 68 len 8192
27/30 Read block 122 len 32768
28/30 Write block 84 len 28672 data: 15 19 18 16 15 15 16
29/30 Write block 158 len 12288 data: 11 10 6
30/30 Read block 566 len 28672
----------------------------------------------------------------
Crucible gen:128 GIO:true work queues: Upstairs:0 downstairs:27
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1003 Acked Write 5 Done Done Done false
1004 Acked Write 10 Done Done Done false
1005 Acked Read 4 Done Done Done false
1006 Acked Read 1 Done Done Done false
1007 Acked Write 2 Done Done Done false
1008 Acked Read 4 Done Done Done false
1009 Acked Read 7 Done Done Done false
1010 Acked Read 9 Done Done Done false
1011 Acked Write 1 Done Done Done false
1012 Acked Write 10 Done Done Done false
1013 Acked Write 5 Done Done Done false
1014 Acked Write 9 Done Done Done false
1015 Acked Write 6 Done Done Done false
1016 Acked Write 9 Done Done Done false
1017 Acked Read 8 Done Done Done false
1018 Acked Read 6 Done Done Done false
1019 Acked Read 2 Done Done Done false
1020 Acked Write 1 Done Done Done false
1021 Acked Read 2 Done Done Done false
1022 Acked Read 1 Done Done Done false
1023 Acked Read 2 Done Done Done false
1024 Acked Write 2 Done Done Done false
1025 Acked Read 2 Done Done Done false
1026 Acked Read 8 Done Done Done false
1027 Acked Write 7 Done Done Sent false
1028 Acked Write 3 Done Done Sent false
1029 Acked Read 7 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 3 3
Done 27 27 24 78
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1002 1002 1002
Downstairs last five completed: 1002 1001 1000
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:32.837237667Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1694}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 126 126 126 233 233 233 T T T
1 020-039 127 127 127 235 235 235 F F F
2 040-059 127 127 127 235 235 235 F F F
3 060-079 125 125 125 232 232 232 F F F
4 080-099 126 126 126 234 234 234 T T T
5 100-119 126 126 126 233 233 233 T T T
6 120-139 127 127 127 235 235 235 F F F
7 140-159 124 124 124 230 230 230 T T T
8 160-179 126 126 126 233 233 233 T T T
9 180-199 127 127 127 235 235 235 T T T
10 200-219 122 122 122 228 228 228 F F F
11 220-239 125 125 125 232 232 232 F F F
12 240-259 125 125 125 232 232 232 T T T
13 260-279 125 125 125 232 232 232 T T T
14 280-299 125 125 125 232 232 232 F F F
15 300-319 128 128 128 236 236 236 T T T
16 320-339 128 128 128 236 236 236 F F F
17 340-359 126 126 126 233 233 233 T T T
18 360-379 119 119 119 222 222 222 F F F
19 380-399 124 124 124 231 231 231 F F F
20 400-419 124 124 124 231 231 231 F F F
21 420-439 123 123 123 229 229 229 T T T
22 440-459 123 123 123 229 229 229 F F F
23 460-479 120 120 120 223 223 223 F F F
24 480-499 125 125 125 232 232 232 F F F
25 500-519 127 127 127 235 235 235 F F F
26 520-539 127 127 127 235 235 235 F F F
27 540-559 112 112 112 205 205 205 T T T
28 560-579 126 126 126 234 234 234 T T T
29 580-599 127 127 127 235 235 235 T T T
Max gen: 128, Max flush: 236
A Difference in extent metadata is expected here
On loop 64
Verifying data now
No change after verify
Loop: 64 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 129 129 129 237 237 237 F F F
1 020-039 127 127 127 235 235 235 F F F
2 040-059 127 127 127 235 235 235 F F F
3 060-079 125 125 125 232 232 232 F F F
4 080-099 129 129 129 237 237 237 F F F
5 100-119 129 129 129 237 237 237 F F F
6 120-139 127 127 127 235 235 235 F F F
7 140-159 129 129 129 237 237 237 F F F
8 160-179 129 129 129 237 237 237 F F F
9 180-199 129 129 129 237 237 237 F F F
10 200-219 122 122 122 228 228 228 F F F
11 220-239 125 125 125 232 232 232 F F F
12 240-259 129 129 129 237 237 237 F F F
13 260-279 129 129 129 237 237 237 F F F
14 280-299 125 125 125 232 232 232 F F F
15 300-319 129 129 129 237 237 237 F F F
16 320-339 128 128 128 236 236 236 F F F
17 340-359 129 129 129 237 237 237 F F F
18 360-379 119 119 119 222 222 222 F F F
19 380-399 124 124 124 231 231 231 F F F
20 400-419 124 124 124 231 231 231 F F F
21 420-439 129 129 129 237 237 237 F F F
22 440-459 123 123 123 229 229 229 F F F
23 460-479 120 120 120 223 223 223 F F F
24 480-499 125 125 125 232 232 232 F F F
25 500-519 127 127 127 235 235 235 F F F
26 520-539 127 127 127 235 235 235 F F F
27 540-559 129 129 129 237 237 237 F F F
28 560-579 129 129 129 237 237 237 F F F
29 580-599 129 129 129 237 237 237 F F F
Max gen: 129, Max flush: 237
Begin loop 65
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.172808423Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.217402349Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.21833676Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.218382902Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.218405368Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.218421078Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702}
{"msg":"Crucible 0aadd02d-aed6-4029-bd88-db3031b71bd7 has session id: 0e963289-8a65-4464-b8b0-004a88805bbc","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219025764Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"Upstairs opts: Upstairs UUID: 0aadd02d-aed6-4029-bd88-db3031b71bd7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219089135Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"Crucible stats registered with UUID: 0aadd02d-aed6-4029-bd88-db3031b71bd7","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219115888Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:37.219136645Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219608443Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"task":"crutest"}
{"msg":"The guest has requested activation with gen:130","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219657263Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702}
{"msg":"0aadd02d-aed6-4029-bd88-db3031b71bd7 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219754463Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219786972Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"0","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219812746Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"0","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219826288Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"1","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.21983983Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"1","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219852552Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"2","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.21987072Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"2","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219886331Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"io task","client":"2","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219910926Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"io task","client":"0","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219934641Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"io task","client":"1","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49966)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.219972937Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"io task","client":"2","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"ds_connection connected from Ok(127.0.0.1:37987)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.220203964Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"io task","client":"0","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"ds_connection connected from Ok(127.0.0.1:38759)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.220368063Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"io task","client":"1","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.220841849Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"2","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.221040226Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"0","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.221540187Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"2","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.221723563Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"0","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.222307122Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.222466093Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.223236756Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"1","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.223875471Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"1","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"[0]R flush_numbers[0..12]: [237, 235, 235, 232, 237, 237, 235, 237, 237, 237, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224447876Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"downstairs","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"[0]R generation[0..12]: [129, 127, 127, 125, 129, 129, 127, 129, 129, 129, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224478317Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"downstairs","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224496836Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"downstairs","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"[1]R flush_numbers[0..12]: [237, 235, 235, 232, 237, 237, 235, 237, 237, 237, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224514115Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"downstairs","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"[1]R generation[0..12]: [129, 127, 127, 125, 129, 129, 127, 129, 129, 129, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224530875Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"downstairs","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224547734Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"downstairs","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"[2]R flush_numbers[0..12]: [237, 235, 235, 232, 237, 237, 235, 237, 237, 237, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224564434Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"downstairs","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"[2]R generation[0..12]: [129, 127, 127, 125, 129, 129, 127, 129, 129, 129, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224581074Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"downstairs","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224597693Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"downstairs","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"Max found gen is 130","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224613364Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"downstairs","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"Generation requested: 130 >= found:130","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224629414Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"downstairs","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"Next flush: 238","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224645254Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","Wait for a query_work_queue command to finish before sending IO
pid":1702,"":"downstairs","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224673147Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"":"downstairs","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224694963Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224710374Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"0","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224726604Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"1","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224742274Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"client":"2","":"downstairs","session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224758654Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224773604Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"0aadd02d-aed6-4029-bd88-db3031b71bd7 is now active with session: 0e963289-8a65-4464-b8b0-004a88805bbc","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224789035Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224803366Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702,"session_id":"0e963289-8a65-4464-b8b0-004a88805bbc"}
{"msg":"The guest has finished waiting for activation with:130","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:37.224818077Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 373 len 4096 data: 12
02/30 Read block 493 len 36864
03/30 Write block 325 len 28672 data: 14 12 15 10 9 8 7
04/30 Read block 593 len 8192
05/30 Write block 506 len 8192 data: 7 7
06/30 Flush
07/30 Write block 331 len 4096 data: 8
08/30 Flush
09/30 Write block 53 len 20480 data: 8 9 10 10 10
10/30 Read block 478 len 24576
11/30 Read block 536 len 40960
12/30 Write block 4 len 24576 data: 8 9 9 11 8 9
13/30 Write block 62 len 20480 data: 11 12 12 15 15
14/30 Flush
15/30 Write block 348 len 8192 data: 15 17
16/30 Read block 469 len 24576
17/30 Read block 205 len 28672
18/30 Read block 275 len 32768
19/30 Flush
20/30 Write block 389 len 20480 data: 11 12 13 13 13
21/30 Write block 517 len 4096 data: 7
22/30 Read block 100 len 12288
23/30 Write block 539 len 32768 data: 8 7 7 6 9 9 8 9
24/30 Read block 3 len 20480
25/30 Read block 245 len 20480
26/30 Read block 240 len 40960
27/30 Read block 67 len 40960
28/30 Write block 575 len 28672 data: 11 9 9 7 7 8 7
29/30 Write block 113 len 24576 data: 10 8 11 15 17 14
30/30 Write block 444 len 12288 data: 6 8 8
----------------------------------------------------------------
Crucible gen:130 GIO:true work queues: Upstairs:0 downstairs:11
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1019 Acked Write 5 Done Done Done false
1020 Acked Write 1 Done Done Done false
1021 Acked Read 3 Done Done Done false
1022 Acked Write 8 Done Done Done false
1023 Acked Read 5 Done Done Done false
1024 Acked Read 5 Done Done Done false
1025 Acked Read 10 Done Done Done false
1026 Acked Read 10 Done Done Done false
1027 Acked Write 7 Sent Sent Sent false
1028 Acked Write 6 Sent Sent Sent false
1029 Acked Write 3 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 3 3 3 9
Done 8 8 8 24
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1018 1018 1018
Downstairs last five completed: 1018 1017 1016 1015 1014
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:37.246400522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1702}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 130 130 130 240 240 240 F F F
1 020-039 127 127 127 235 235 235 F F F
2 040-059 130 130 130 240 240 240 F F F
3 060-079 130 130 130 240 240 240 F F F
4 080-099 129 129 129 237 237 237 F F F
5 100-119 129 129 129 237 237 237 T T T
6 120-139 127 127 127 235 235 235 F F F
7 140-159 129 129 129 237 237 237 F F F
8 160-179 129 129 129 237 237 237 F F F
9 180-199 129 129 129 237 237 237 F F F
10 200-219 122 122 122 228 228 228 F F F
11 220-239 125 125 125 232 232 232 F F F
12 240-259 129 129 129 237 237 237 F F F
13 260-279 129 129 129 237 237 237 F F F
14 280-299 125 125 125 232 232 232 F F F
15 300-319 129 129 129 237 237 237 F F F
16 320-339 130 130 130 239 239 239 F F F
17 340-359 130 130 130 241 241 241 F F F
18 360-379 130 130 130 238 238 238 F F F
19 380-399 124 124 124 231 231 231 T T T
20 400-419 124 124 124 231 231 231 F F F
21 420-439 129 129 129 237 237 237 F F F
22 440-459 123 123 123 229 229 229 T T T
23 460-479 120 120 120 223 223 223 F F F
24 480-499 125 125 125 232 232 232 F F F
25 500-519 130 130 130 238 238 238 T T T
26 520-539 127 127 127 235 235 235 T T T
27 540-559 129 129 129 237 237 237 T T T
28 560-579 129 129 129 237 237 237 T T T
29 580-599 129 129 129 237 237 237 T T T
Max gen: 130, Max flush: 241
A Difference in extent metadata is expected here
On loop 65
Verifying data now
No change after verify
Loop: 65 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 130 130 130 240 240 240 F F F
1 020-039 127 127 127 235 235 235 F F F
2 040-059 130 130 130 240 240 240 F F F
3 060-079 130 130 130 240 240 240 F F F
4 080-099 129 129 129 237 237 237 F F F
5 100-119 131 131 131 242 242 242 F F F
6 120-139 127 127 127 235 235 235 F F F
7 140-159 129 129 129 237 237 237 F F F
8 160-179 129 129 129 237 237 237 F F F
9 180-199 129 129 129 237 237 237 F F F
10 200-219 122 122 122 228 228 228 F F F
11 220-239 125 125 125 232 232 232 F F F
12 240-259 129 129 129 237 237 237 F F F
13 260-279 129 129 129 237 237 237 F F F
14 280-299 125 125 125 232 232 232 F F F
15 300-319 129 129 129 237 237 237 F F F
16 320-339 130 130 130 239 239 239 F F F
17 340-359 130 130 130 241 241 241 F F F
18 360-379 130 130 130 238 238 238 F F F
19 380-399 131 131 131 242 242 242 F F F
20 400-419 124 124 124 231 231 231 F F F
21 420-439 129 129 129 237 237 237 F F F
22 440-459 131 131 131 242 242 242 F F F
23 460-479 120 120 120 223 223 223 F F F
24 480-499 125 125 125 232 232 232 F F F
25 500-519 131 131 131 242 242 242 F F F
26 520-539 131 131 131 242 242 242 F F F
27 540-559 131 131 131 242 242 242 F F F
28 560-579 131 131 131 242 242 242 F F F
29 580-599 131 131 131 242 242 242 F F F
Max gen: 131, Max flush: 242
Begin loop 66
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.516849834Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.561677585Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.56247671Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.562523331Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.562537482Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.562551524Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710}
{"msg":"Crucible 2c4d7c5c-52ff-4e42-b8f0-4dee3eb1e6d7 has session id: e478f637-4396-4a5d-8be8-fc44fc5276e1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.5630586Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"Upstairs opts: Upstairs UUID: 2c4d7c5c-52ff-4e42-b8f0-4dee3eb1e6d7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563103022Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"Crucible stats registered with UUID: 2c4d7c5c-52ff-4e42-b8f0-4dee3eb1e6d7","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563120372Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:41.563133293Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563627038Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"task":"crutest"}
{"msg":"The guest has requested activation with gen:132","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563652442Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710}
{"msg":"2c4d7c5c-52ff-4e42-b8f0-4dee3eb1e6d7 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563732932Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563750841Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"0","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563774756Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"0","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563788138Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"1","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563801819Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"1","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563814841Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"2","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563828533Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"2","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563841475Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563855306Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"io task","client":"0","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.563908663Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"io task","client":"1","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:41.56407593Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52527)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.564134364Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"io task","client":"0","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"ds_connection connected from Ok(127.0.0.1:51386)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.564174059Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"io task","client":"1","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:41.564215553Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:41.564256528Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.564468267Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.564925574Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"1","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.564967577Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"0","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.565645957Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"0","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.565679756Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"1","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.566408256Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:41.566440556Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.56580457Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53147)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.565882991Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"io task","client":"2","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.567117517Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"2","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.567669135Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"2","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"[0]R flush_numbers[0..12]: [240, 235, 240, 240, 237, 242, 235, 237, 237, 237, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568293269Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"downstairs","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"[0]R generation[0..12]: [130, 127, 130, 130, 129, 131, 127, 129, 129, 129, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568318723Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"downstairs","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568335263Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"downstairs","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"[1]R flush_numbers[0..12]: [240, 235, 240, 240, 237, 242, 235, 237, 237, 237, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568356959Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"downstairs","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"[1]R generation[0..12]: [130, 127, 130, 130, 129, 131, 127, 129, 129, 129, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.56837234Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"downstairs","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.56838753Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"downstairs","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"[2]R flush_numbers[0..12]: [240, 235, 240, 240, 237, 242, 235, 237, 237, 237, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568402501Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"downstairs","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"[2]R generation[0..12]: [130, 127, 130, 130, 129, 131, 127, 129, 129, 129, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568417412Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"downstairs","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568432602Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"downstairs","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"Max found gen is 132","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568446714Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"downstairs","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"Generation requested: 132 >= found:132","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568461494Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"downstairs","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"Next flush: 243","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568475665Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"downstairs","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568489707Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"":"downstairs","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:41:51.568503808Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568522826Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"0","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568537018Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"1","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568551139Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"client":"2","":"downstairs","session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568568068Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.56858124Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"2c4d7c5c-52ff-4e42-b8f0-4dee3eb1e6d7 is now active with session: e478f637-4396-4a5d-8be8-fc44fc5276e1","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568595131Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568608034Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710,"session_id":"e478f637-4396-4a5d-8be8-fc44fc5276e1"}
{"msg":"The guest has finished waiting for activation with:132","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:51.568621166Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 95 len 16384 data: 14 12 11 13
02/30 Write block 473 len 40960 data: 9 9 9 10 11 13 10 11 10 10
03/30 Write block 359 len 16384 data: 13 12 10 9
04/30 Read block 158 len 32768
05/30 Flush
06/30 Write block 68 len 36864 data: 15 12 14 15 10 12 11 10 7
07/30 Flush
08/30 Write block 561 len 40960 data: 7 6 6 6 7 7 7 8 7 9
09/30 Write block 536 len 8192 data: 11 9
10/30 Read block 346 len 32768
11/30 Write block 54 len 32768 data: 10 11 11 11 10 9 9 9
12/30 Read block 241 len 4096
13/30 Read block 115 len 4096
14/30 Read block 14 len 36864
15/30 Read block 437 len 16384
16/30 Read block 374 len 24576
17/30 Read block 179 len 4096
18/30 Read block 322 len 36864
19/30 Write block 410 len 16384 data: 9 9 7 7
20/30 Read block 136 len 20480
21/30 Write block 525 len 8192 data: 12 10
22/30 Write block 533 len 12288 data: 13 12 14
23/30 Write block 256 len 32768 data: 11 12 10 9 11 12 11 11
24/30 Write block 304 len 20480 data: 11 12 14 14 14
25/30 Read block 295 len 12288
26/30 Read block 69 len 8192
27/30 Write block 590 len 32768 data: 10 9 9 10 8 8 10 6
28/30 Read block 174 len 4096
29/30 Write block 176 len 12288 data: 12 11 12
30/30 Write block 264 len 4096 data: 12
----------------------------------------------------------------
Crucible gen:132 GIO:true work queues: Upstairs:0 downstairs:23
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1007 Acked Write 10 Done Done Done false
1008 Acked Write 2 Done Done Done false
1009 Acked Read 8 Done Done Done false
1010 Acked Write 8 Done Done Done false
1011 Acked Read 1 Done Done Done false
1012 Acked Read 1 Done Done Done false
1013 Acked Read 9 Done Done Done false
1014 Acked Read 4 Done Done Done false
1015 Acked Read 6 Done Done Done false
1016 Acked Read 1 Done Done Done false
1017 Acked Read 9 Done Done Done false
1018 Acked Write 4 Done Done Done false
1019 Acked Read 5 Done Done Done false
1020 Acked Write 2 Done Done Done false
1021 Acked Write 3 Done Done Done false
1022 Acked Write 8 Done Done Done false
1023 Acked Write 5 Done Done Done false
1024 Acked Read 3 Done Done Done false
1025 Acked Read 2 Done Done Done false
1026 Acked Write 8 Done Done Done false
1027 Acked Read 1 Done Done Done false
1028 Acked Write 3 Sent Sent Sent false
1029 Acked Write 1 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 21 21 21 63
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1006 1006 1006
Downstairs last five completed: 1006 1005 1004 1003 1002
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:51.588577887Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1710}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 130 130 130 240 240 240 F F F
1 020-039 127 127 127 235 235 235 F F F
2 040-059 130 130 130 240 240 240 T T T
3 060-079 132 132 132 244 244 244 T T T
4 080-099 132 132 132 243 243 243 F F F
5 100-119 131 131 131 242 242 242 F F F
6 120-139 127 127 127 235 235 235 F F F
7 140-159 129 129 129 237 237 237 F F F
8 160-179 129 129 129 237 237 237 T T T
9 180-199 129 129 129 237 237 237 F F F
10 200-219 122 122 122 228 228 228 F F F
11 220-239 125 125 125 232 232 232 F F F
12 240-259 129 129 129 237 237 237 T T T
13 260-279 129 129 129 237 237 237 T T T
14 280-299 125 125 125 232 232 232 F F F
15 300-319 129 129 129 237 237 237 T T T
16 320-339 130 130 130 239 239 239 F F F
17 340-359 132 132 132 243 243 243 F F F
18 360-379 132 132 132 243 243 243 F F F
19 380-399 131 131 131 242 242 242 F F F
20 400-419 124 124 124 231 231 231 T T T
21 420-439 129 129 129 237 237 237 F F F
22 440-459 131 131 131 242 242 242 F F F
23 460-479 132 132 132 243 243 243 F F F
24 480-499 132 132 132 243 243 243 F F F
25 500-519 131 131 131 242 242 242 F F F
26 520-539 131 131 131 242 242 242 T T T
27 540-559 131 131 131 242 242 242 F F F
28 560-579 131 131 131 242 242 242 T T T
29 580-599 131 131 131 242 242 242 T T T
Max gen: 132, Max flush: 244
A Difference in extent metadata is expected here
On loop 66
Verifying data now
No change after verify
Loop: 66 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 130 130 130 240 240 240 F F F
1 020-039 127 127 127 235 235 235 F F F
2 040-059 133 133 133 245 245 245 F F F
3 060-079 133 133 133 245 245 245 F F F
4 080-099 132 132 132 243 243 243 F F F
5 100-119 131 131 131 242 242 242 F F F
6 120-139 127 127 127 235 235 235 F F F
7 140-159 129 129 129 237 237 237 F F F
8 160-179 133 133 133 245 245 245 F F F
9 180-199 129 129 129 237 237 237 F F F
10 200-219 122 122 122 228 228 228 F F F
11 220-239 125 125 125 232 232 232 F F F
12 240-259 133 133 133 245 245 245 F F F
13 260-279 133 133 133 245 245 245 F F F
14 280-299 125 125 125 232 232 232 F F F
15 300-319 133 133 133 245 245 245 F F F
16 320-339 130 130 130 239 239 239 F F F
17 340-359 132 132 132 243 243 243 F F F
18 360-379 132 132 132 243 243 243 F F F
19 380-399 131 131 131 242 242 242 F F F
20 400-419 133 133 133 245 245 245 F F F
21 420-439 129 129 129 237 237 237 F F F
22 440-459 131 131 131 242 242 242 F F F
23 460-479 132 132 132 243 243 243 F F F
24 480-499 132 132 132 243 243 243 F F F
25 500-519 131 131 131 242 242 242 F F F
26 520-539 133 133 133 245 245 245 F F F
27 540-559 131 131 131 242 242 242 F F F
28 560-579 133 133 133 245 245 245 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 133, Max flush: 245
Begin loop 67
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.867060265Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.911802509Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.912921556Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.912966258Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.912981588Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.913002845Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718}
{"msg":"Crucible eb6e3a2d-9139-425b-afba-6de288c3debb has session id: 1e3aed97-64be-4540-9010-19ae6d8a7acc","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.913830773Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"Upstairs opts: Upstairs UUID: eb6e3a2d-9139-425b-afba-6de288c3debb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.91386765Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"Crucible stats registered with UUID: eb6e3a2d-9139-425b-afba-6de288c3debb","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.91389929Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:55.913924834Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.9144332Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"task":"crutest"}
{"msg":"The guest has requested activation with gen:134","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.914458804Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718}
{"msg":"eb6e3a2d-9139-425b-afba-6de288c3debb active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.914542152Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.914576561Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"0","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.914606132Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"0","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.914627669Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"1","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.914728697Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"1","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.914770021Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"2","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.914799193Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"2","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.914824817Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"io task","client":"2","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.914856527Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"io task","client":"0","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.914879453Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"io task","client":"1","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"ds_connection connected from Ok(127.0.0.1:54869)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.914902559Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"io task","client":"1","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"ds_connection connected from Ok(127.0.0.1:64724)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.914926994Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"io task","client":"2","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"ds_connection connected from Ok(127.0.0.1:51378)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.915113998Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"io task","client":"0","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.91577275Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"2","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.915813325Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"0","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.915855069Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"1","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.916572015Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"1","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.916600407Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"2","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.916615188Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"0","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917276159Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.91741919Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"[0]R flush_numbers[0..12]: [240, 235, 245, 245, 243, 242, 235, 237, 245, 237, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917490296Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"downstairs","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"[0]R generation[0..12]: [130, 127, 133, 133, 132, 131, 127, 129, 133, 129, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917511043Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"downstairs","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917525185Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"downstairs","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"[1]R flush_numbers[0..12]: [240, 235, 245, 245, 243, 242, 235, 237, 245, 237, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917539826Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"downstairs","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"[1]R generation[0..12]: [130, 127, 133, 133, 132, 131, 127, 129, 133, 129, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917553467Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"downstairs","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917567229Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"downstairs","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"[2]R flush_numbers[0..12]: [240, 235, 245, 245, 243, 242, 235, 237, 245, 237, 228, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.91758089Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"downstairs","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"[2]R generation[0..12]: [130, 127, 133, 133, 132, 131, 127, 129, 133, 129, 122, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917594252Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"downstairs","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917611132Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"downstairs","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"Max found gen is 134","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917624153Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"downstairs","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"Generation requested: 134 >= found:134","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917637125Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"downstairs","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"Next flush: 246","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:41:55.917649728Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"downstairs","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917666127Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"":"downstairs","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917678919Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917690443Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"0","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917703055Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"1","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917715767Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"client":"2","":"downstairs","session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917728409Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917740172Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"eb6e3a2d-9139-425b-afba-6de288c3debb is now active with session: 1e3aed97-64be-4540-9010-19ae6d8a7acc","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917752414Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.917764067Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718,"session_id":"1e3aed97-64be-4540-9010-19ae6d8a7acc"}
{"msg":"The guest has finished waiting for activation with:134","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:41:55.91777589Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 207 len 4096 data: 12
02/30 Read block 366 len 4096
03/30 Read block 418 len 4096
04/30 Flush
05/30 Write block 41 len 28672 data: 11 9 8 8 9 9 7
06/30 Flush
07/30 Flush
08/30 Write block 21 len 16384 data: 9 12 14 16
09/30 Write block 471 len 4096 data: 9
10/30 Read block 345 len 28672
11/30 Read block 210 len 20480
12/30 Read block 440 len 4096
13/30 Read block 256 len 40960
14/30 Read block 226 len 24576
15/30 Write block 435 len 12288 data: 16 13 7
16/30 Read block 505 len 8192
17/30 Write block 362 len 36864 data: 10 8 7 6 6 6 9 11 12
18/30 Write block 480 len 16384 data: 12 11 11 11
19/30 Read block 127 len 40960
20/30 Read block 197 len 16384
21/30 Read block 496 len 28672
22/30 Write block 434 len 36864 data: 14 17 14 8 7 7 9 9 9
23/30 Write block 373 len 8192 data: 13 15
24/30 Read block 326 len 40960
25/30 Read block 254 len 20480
26/30 Write block 368 len 24576 data: 10 12 13 13 11 14
27/30 Read block 290 len 36864
28/30 Read block 529 len 4096
29/30 Read block 432 len 40960
30/30 Read block 425 len 4096
----------------------------------------------------------------
Crucible gen:134 GIO:true work queues: Upstairs:0 downstairs:23
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1007 Acked Write 4 Done Done Done false
1008 Acked Write 1 Done Done Done false
1009 Acked Read 7 Done Done Done false
1010 Acked Read 5 Done Done Done false
1011 Acked Read 1 Done Done Done false
1012 Acked Read 10 Done Done Done false
1013 Acked Read 6 Done Done Done false
1014 Acked Write 3 Done Done Done false
1015 Acked Read 2 Done Done Done false
1016 Acked Write 9 Done Done Done false
1017 Acked Write 4 Done Done Done false
1018 Acked Read 10 Done Done Done false
1019 Acked Read 4 Done Done Done false
1020 Acked Read 7 Done Done Done false
1021 Acked Write 9 Done Done Done false
1022 Acked Write 2 Done Done Done false
1023 Acked Read 10 Done Done Done false
1024 Acked Read 5 Done Done Done false
1025 Acked Write 6 Done Done Done false
1026 Acked Read 9 Done Done Done false
1027 Acked Read 1 Done Done Done false
1028 Acked Read 10 Done Done Done false
1029 Acked Read 1 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 23 23 23 69
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1006 1006 1006
Downstairs last five completed: 1006 1005 1004 1003 1002
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:41:55.938748043Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1718}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 130 130 130 240 240 240 F F F
1 020-039 127 127 127 235 235 235 T T T
2 040-059 134 134 134 247 247 247 F F F
3 060-079 133 133 133 245 245 245 F F F
4 080-099 132 132 132 243 243 243 F F F
5 100-119 131 131 131 242 242 242 F F F
6 120-139 127 127 127 235 235 235 F F F
7 140-159 129 129 129 237 237 237 F F F
8 160-179 133 133 133 245 245 245 F F F
9 180-199 129 129 129 237 237 237 F F F
10 200-219 134 134 134 246 246 246 F F F
11 220-239 125 125 125 232 232 232 F F F
12 240-259 133 133 133 245 245 245 F F F
13 260-279 133 133 133 245 245 245 F F F
14 280-299 125 125 125 232 232 232 F F F
15 300-319 133 133 133 245 245 245 F F F
16 320-339 130 130 130 239 239 239 F F F
17 340-359 132 132 132 243 243 243 F F F
18 360-379 132 132 132 243 243 243 T T T
19 380-399 131 131 131 242 242 242 F F F
20 400-419 133 133 133 245 245 245 F F F
21 420-439 129 129 129 237 237 237 T T T
22 440-459 131 131 131 242 242 242 T T T
23 460-479 132 132 132 243 243 243 T T T
24 480-499 132 132 132 243 243 243 T T T
25 500-519 131 131 131 242 242 242 F F F
26 520-539 133 133 133 245 245 245 F F F
27 540-559 131 131 131 242 242 242 F F F
28 560-579 133 133 133 245 245 245 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 134, Max flush: 247
A Difference in extent metadata is expected here
On loop 67
Verifying data now
No change after verify
Loop: 67 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 130 130 130 240 240 240 F F F
1 020-039 135 135 135 248 248 248 F F F
2 040-059 134 134 134 247 247 247 F F F
3 060-079 133 133 133 245 245 245 F F F
4 080-099 132 132 132 243 243 243 F F F
5 100-119 131 131 131 242 242 242 F F F
6 120-139 127 127 127 235 235 235 F F F
7 140-159 129 129 129 237 237 237 F F F
8 160-179 133 133 133 245 245 245 F F F
9 180-199 129 129 129 237 237 237 F F F
10 200-219 134 134 134 246 246 246 F F F
11 220-239 125 125 125 232 232 232 F F F
12 240-259 133 133 133 245 245 245 F F F
13 260-279 133 133 133 245 245 245 F F F
14 280-299 125 125 125 232 232 232 F F F
15 300-319 133 133 133 245 245 245 F F F
16 320-339 130 130 130 239 239 239 F F F
17 340-359 132 132 132 243 243 243 F F F
18 360-379 135 135 135 248 248 248 F F F
19 380-399 131 131 131 242 242 242 F F F
20 400-419 133 133 133 245 245 245 F F F
21 420-439 135 135 135 248 248 248 F F F
22 440-459 135 135 135 248 248 248 F F F
23 460-479 135 135 135 248 248 248 F F F
24 480-499 135 135 135 248 248 248 F F F
25 500-519 131 131 131 242 242 242 F F F
26 520-539 133 133 133 245 245 245 F F F
27 540-559 131 131 131 242 242 242 F F F
28 560-579 133 133 133 245 245 245 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 135, Max flush: 248
Begin loop 68
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.195105721Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:00.239462483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.240137635Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.272258094Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.273311032Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.273352087Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.273371715Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.273387075Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726}
{"msg":"Crucible 48624471-5137-440f-ae6c-5dd8c120b48b has session id: 3da4b347-2529-4da4-bd3b-7d16db3763ff","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.27396078Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"Upstairs opts: Upstairs UUID: 48624471-5137-440f-ae6c-5dd8c120b48b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.274049015Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"Crucible stats registered with UUID: 48624471-5137-440f-ae6c-5dd8c120b48b","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.274071441Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:00.274085113Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.274557271Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"task":"crutest"}
{"msg":"The guest has requested activation with gen:136","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.274589401Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726}
{"msg":"48624471-5137-440f-ae6c-5dd8c120b48b active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.274675058Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.274692537Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"0","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.274721609Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"0","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.27473561Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"1","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.274759875Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"1","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.274787478Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"2","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.274820728Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"2","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.27484836Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"io task","client":"2","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.274878122Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"io task","client":"0","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.274907314Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"io task","client":"1","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"ds_connection connected from Ok(127.0.0.1:63548)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.275011939Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"io task","client":"1","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53645)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.275062668Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"io task","client":"2","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"ds_connection connected from Ok(127.0.0.1:34615)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.275121961Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"io task","client":"0","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.275899969Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"2","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.275944841Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"1","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.276208528Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"0","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.27672252Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"1","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.27675547Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"2","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277091581Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"0","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277413522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277446522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"[0]R flush_numbers[0..12]: [240, 248, 247, 245, 243, 242, 235, 237, 245, 237, 246, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277737741Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"downstairs","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"[0]R generation[0..12]: [130, 135, 134, 133, 132, 131, 127, 129, 133, 129, 134, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277766384Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"downstairs","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277781574Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"downstairs","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"[1]R flush_numbers[0..12]: [240, 248, 247, 245, 243, 242, 235, 237, 245, 237, 246, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277795386Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"downstairs","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"[1]R generation[0..12]: [130, 135, 134, 133, 132, 131, 127, 129, 133, 129, 134, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277808947Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"downstairs","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277823208Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"downstairs","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"[2]R flush_numbers[0..12]: [240, 248, 247, 245, 243, 242, 235, 237, 245, 237, 246, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.27783667Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"downstairs","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"[2]R generation[0..12]: [130, 135, 134, 133, 132, 131, 127, 129, 133, 129, 134, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277850241Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"downstairs","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:42:00.277863683Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"downstairs","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"Max found gen is 136","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277883811Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"downstairs","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"Generation requested: 136 >= found:136","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277896883Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"downstairs","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"Next flush: 249","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277909445Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"downstairs","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277921917Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"":"downstairs","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277934589Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277950659Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"0","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277963761Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"1","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277976363Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"client":"2","":"downstairs","session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.277989106Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.278000908Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"48624471-5137-440f-ae6c-5dd8c120b48b is now active with session: 3da4b347-2529-4da4-bd3b-7d16db3763ff","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.278013311Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.278024773Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726,"session_id":"3da4b347-2529-4da4-bd3b-7d16db3763ff"}
{"msg":"The guest has finished waiting for activation with:136","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:00.278037106Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 416 len 36864 data: 9 11 12 13 13 11 12 13 10
02/30 Read block 511 len 20480
03/30 Read block 140 len 20480
04/30 Read block 466 len 36864
05/30 Write block 62 len 12288 data: 12 13 13
06/30 Write block 285 len 40960 data: 6 5 6 6 8 8 11 13 14 12
07/30 Write block 402 len 16384 data: 13 11 11 12
08/30 Flush
09/30 Write block 194 len 8192 data: 10 14
10/30 Read block 112 len 28672
11/30 Write block 28 len 16384 data: 10 10 12 10
12/30 Flush
13/30 Write block 278 len 36864 data: 16 14 14 13 11 7 6 7 6
14/30 Flush
15/30 Write block 537 len 28672 data: 10 8 9 8 8 7 10
16/30 Read block 169 len 32768
17/30 Write block 316 len 24576 data: 9 7 6 8 9 9
18/30 Flush
19/30 Write block 155 len 40960 data: 10 11 11 12 11 7 12 10 8 12
20/30 Write block 569 len 32768 data: 8 10 9 10 11 10 12 10
21/30 Read block 7 len 40960
22/30 Write block 556 len 40960 data: 13 12 11 12 11 8 7 7 7 8
23/30 Write block 27 len 28672 data: 11 11 11 13 11 9 9
24/30 Write block 120 len 4096 data: 13
25/30 Write block 102 len 36864 data: 15 16 18 17 15 11 11 11 11
26/30 Write block 70 len 28672 data: 15 16 11 13 12 11 8
27/30 Read block 254 len 36864
28/30 Write block 564 len 4096 data: 8
29/30 Write block 541 len 12288 data: 9 8 11
30/30 Write block 93 len 4096 data: 19
----------------------------------------------------------------
Crucible gen:136 GIO:true work queues: Upstairs:0 downstairs:12
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1018 Acked Write 10 Done Done Done false
1019 Acked Write 8 Done Done Done false
1020 Acked Read 10 Done Done Done false
1021 Acked Write 10 Done Done Done false
1022 Acked Write 7 Done Done Done false
1023 Acked Write 1 Done Done Done false
1024 Acked Write 9 Done Done Done false
1025 Acked Write 7 Done Done Done false
1026 Acked Read 9 Done Done Done false
1027 Acked Write 1 Done Done Done false
1028 Acked Write 3 Sent Sent Sent false
1029 Acked Write 1 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 10 10 10 30
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1017 1017 1017
Downstairs last five completed: 1017 1016 1015 1014 1013
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:00.298764041Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1726}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 130 130 130 240 240 240 F F F
1 020-039 136 136 136 250 250 250 T T T
2 040-059 134 134 134 247 247 247 F F F
3 060-079 136 136 136 249 249 249 T T T
4 080-099 132 132 132 243 243 243 T T T
5 100-119 131 131 131 242 242 242 T T T
6 120-139 127 127 127 235 235 235 T T T
7 140-159 129 129 129 237 237 237 T T T
8 160-179 133 133 133 245 245 245 T T T
9 180-199 136 136 136 250 250 250 F F F
10 200-219 134 134 134 246 246 246 F F F
11 220-239 125 125 125 232 232 232 F F F
12 240-259 133 133 133 245 245 245 F F F
13 260-279 136 136 136 251 251 251 F F F
14 280-299 136 136 136 251 251 251 F F F
15 300-319 136 136 136 252 252 252 F F F
16 320-339 136 136 136 252 252 252 F F F
17 340-359 132 132 132 243 243 243 F F F
18 360-379 135 135 135 248 248 248 F F F
19 380-399 131 131 131 242 242 242 F F F
20 400-419 136 136 136 249 249 249 F F F
21 420-439 136 136 136 249 249 249 F F F
22 440-459 135 135 135 248 248 248 F F F
23 460-479 135 135 135 248 248 248 F F F
24 480-499 135 135 135 248 248 248 F F F
25 500-519 131 131 131 242 242 242 F F F
26 520-539 136 136 136 252 252 252 F F F
27 540-559 136 136 136 252 252 252 T T T
28 560-579 133 133 133 245 245 245 T T T
29 580-599 133 133 133 245 245 245 F F F
Max gen: 136, Max flush: 252
A Difference in extent metadata is expected here
On loop 68
Verifying data now
No change after verify
Loop: 68 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 130 130 130 240 240 240 F F F
1 020-039 137 137 137 253 253 253 F F F
2 040-059 134 134 134 247 247 247 F F F
3 060-079 137 137 137 253 253 253 F F F
4 080-099 137 137 137 253 253 253 F F F
5 100-119 137 137 137 253 253 253 F F F
6 120-139 137 137 137 253 253 253 F F F
7 140-159 137 137 137 253 253 253 F F F
8 160-179 137 137 137 253 253 253 F F F
9 180-199 136 136 136 250 250 250 F F F
10 200-219 134 134 134 246 246 246 F F F
11 220-239 125 125 125 232 232 232 F F F
12 240-259 133 133 133 245 245 245 F F F
13 260-279 136 136 136 251 251 251 F F F
14 280-299 136 136 136 251 251 251 F F F
15 300-319 136 136 136 252 252 252 F F F
16 320-339 136 136 136 252 252 252 F F F
17 340-359 132 132 132 243 243 243 F F F
18 360-379 135 135 135 248 248 248 F F F
19 380-399 131 131 131 242 242 242 F F F
20 400-419 136 136 136 249 249 249 F F F
21 420-439 136 136 136 249 249 249 F F F
22 440-459 135 135 135 248 248 248 F F F
23 460-479 135 135 135 248 248 248 F F F
24 480-499 135 135 135 248 248 248 F F F
25 500-519 131 131 131 242 242 242 F F F
26 520-539 136 136 136 252 252 252 F F F
27 540-559 137 137 137 253 253 253 F F F
28 560-579 137 137 137 253 253 253 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 137, Max flush: 253
Begin loop 69
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.603622724Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.648755336Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.649578586Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.649638459Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.649659956Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.649681993Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734}
{"msg":"Crucible 35a3d69c-e084-4766-b28f-1d3e0b34604e has session id: c87ede9b-4ffc-4d34-aaa7-653236bc533a","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.650311363Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"Upstairs opts: Upstairs UUID: 35a3d69c-e084-4766-b28f-1d3e0b34604e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.650437225Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"Crucible stats registered with UUID: 35a3d69c-e084-4766-b28f-1d3e0b34604e","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.650465557Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:04.650479179Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.650926492Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"task":"crutest"}
{"msg":"The guest has requested activation with gen:138","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.650960471Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734}
{"msg":"35a3d69c-e084-4766-b28f-1d3e0b34604e active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.651022752Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.651049955Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"0","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.651075719Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"0","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.65109013Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"1","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.65110652Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"1","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.651119692Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"2","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.651133414Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"2","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.651146226Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"io task","client":"2","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.651159638Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"io task","client":"0","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.651173019Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"io task","client":"1","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"ds_connection connected from Ok(127.0.0.1:63743)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.65126824Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"io task","client":"2","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50439)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.651315681Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"io task","client":"1","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"ds_connection connected from Ok(127.0.0.1:58772)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.65134881Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"io task","client":"0","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.652109269Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"2","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.652154302Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"0","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.65280257Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"0","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.652973474Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"2","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.653447281Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.653528271Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.653607332Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"1","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.654174051Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"1","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"[0]R flush_numbers[0..12]: [240, 253, 247, 253, 253, 253, 253, 253, 253, 250, 246, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.654713777Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"downstairs","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"[0]R generation[0..12]: [130, 137, 134, 137, 137, 137, 137, 137, 137, 136, 134, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.654738261Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"downstairs","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.65475646Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"downstairs","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"[1]R flush_numbers[0..12]: [240, 253, 247, 253, 253, 253, 253, 253, 253, 250, 246, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.654774109Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"downstairs","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"[1]R generation[0..12]: [130, 137, 134, 137, 137, 137, 137, 137, 137, 136, 134, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.654791418Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"downstairs","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.654809277Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"downstairs","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"[2]R flush_numbers[0..12]: [240, 253, 247, 253, 253, 253, 253, 253, 253, 250, 246, 232]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.654826237Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"downstairs","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"[2]R generation[0..12]: [130, 137, 134, 137, 137, 137, 137, 137, 137, 136, 134, 125]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.654848173Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"downstairs","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.654865683Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":Wait for a query_work_queue command to finish before sending IO
1734,"":"downstairs","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"Max found gen is 138","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.654896823Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"downstairs","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"Generation requested: 138 >= found:138","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.654927854Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"downstairs","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"Next flush: 254","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.654961263Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"downstairs","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.654992064Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"":"downstairs","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.655024604Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.655055425Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"0","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.655090093Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"1","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.655123443Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"client":"2","":"downstairs","session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
,"time":"2025-01-27T19:42:04.655157822Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.655187473Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"35a3d69c-e084-4766-b28f-1d3e0b34604e is now active with session: c87ede9b-4ffc-4d34-aaa7-653236bc533a","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.655217654Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.655233944Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734,"session_id":"c87ede9b-4ffc-4d34-aaa7-653236bc533a"}
{"msg":"The guest has finished waiting for activation with:138","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:04.655249235Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 238 len 24576 data: 4 3 5 8 11 13
02/30 Read block 320 len 24576
03/30 Read block 337 len 8192
04/30 Read block 560 len 20480
05/30 Read block 421 len 12288
06/30 Read block 6 len 32768
07/30 Flush
08/30 Write block 226 len 4096 data: 8
09/30 Write block 362 len 28672 data: 11 9 8 7 7 7 11
10/30 Read block 336 len 8192
11/30 Read block 364 len 40960
12/30 Read block 459 len 12288
13/30 Write block 383 len 28672 data: 6 6 6 9 9 10 12
14/30 Flush
15/30 Write block 274 len 16384 data: 15 17 15 16
16/30 Read block 519 len 8192
17/30 Read block 487 len 4096
18/30 Write block 443 len 40960 data: 8 7 9 9 7 6 7 6 7 6
19/30 Read block 399 len 8192
20/30 Write block 496 len 16384 data: 16 13 13 12
21/30 Flush
22/30 Write block 228 len 32768 data: 8 8 12 10 7 7 7 8
23/30 Write block 81 len 40960 data: 11 13 12 16 20 19 17 16 16 17
24/30 Write block 27 len 12288 data: 12 12 12
25/30 Read block 425 len 36864
26/30 Flush
27/30 Write block 325 len 28672 data: 15 13 16 11 10 9 9
28/30 Write block 571 len 32768 data: 10 11 12 11 13 11 10 8
29/30 Read block 420 len 4096
30/30 Write block 449 len 4096 data: 8
----------------------------------------------------------------
Crucible gen:138 GIO:true work queues: Upstairs:0 downstairs:4
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1026 Acked Write 7 Done Done Done false
1027 Acked Write 8 Done Done Done false
1028 Acked Read 1 Done Sent Done false
1029 Acked Write 1 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 2 1 4
Done 3 2 3 8
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1025 1025 1025
Downstairs last five completed: 1025 1024 1023 1022 1021
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:04.67730006Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1734}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 130 130 130 240 240 240 F F F
1 020-039 138 138 138 257 257 257 F F F
2 040-059 134 134 134 247 247 247 F F F
3 060-079 137 137 137 253 253 253 F F F
4 080-099 138 138 138 257 257 257 F F F
5 100-119 137 137 137 253 253 253 F F F
6 120-139 137 137 137 253 253 253 F F F
7 140-159 137 137 137 253 253 253 F F F
8 160-179 137 137 137 253 253 253 F F F
9 180-199 136 136 136 250 250 250 F F F
10 200-219 134 134 134 246 246 246 F F F
11 220-239 138 138 138 257 257 257 F F F
12 240-259 138 138 138 254 254 254 F F F
13 260-279 138 138 138 256 256 256 F F F
14 280-299 136 136 136 251 251 251 F F F
15 300-319 136 136 136 252 252 252 F F F
16 320-339 136 136 136 252 252 252 T T T
17 340-359 132 132 132 243 243 243 F F F
18 360-379 138 138 138 255 255 255 F F F
19 380-399 138 138 138 255 255 255 F F F
20 400-419 136 136 136 249 249 249 F F F
21 420-439 136 136 136 249 249 249 F F F
22 440-459 138 138 138 256 256 256 T T T
23 460-479 135 135 135 248 248 248 F F F
24 480-499 138 138 138 256 256 256 F F F
25 500-519 131 131 131 242 242 242 F F F
26 520-539 136 136 136 252 252 252 F F F
27 540-559 137 137 137 253 253 253 F F F
28 560-579 137 137 137 253 253 253 T T T
29 580-599 133 133 133 245 245 245 F F F
Max gen: 138, Max flush: 257
A Difference in extent metadata is expected here
On loop 69
Verifying data now
No change after verify
Loop: 69 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 130 130 130 240 240 240 F F F
1 020-039 138 138 138 257 257 257 F F F
2 040-059 134 134 134 247 247 247 F F F
3 060-079 137 137 137 253 253 253 F F F
4 080-099 138 138 138 257 257 257 F F F
5 100-119 137 137 137 253 253 253 F F F
6 120-139 137 137 137 253 253 253 F F F
7 140-159 137 137 137 253 253 253 F F F
8 160-179 137 137 137 253 253 253 F F F
9 180-199 136 136 136 250 250 250 F F F
10 200-219 134 134 134 246 246 246 F F F
11 220-239 138 138 138 257 257 257 F F F
12 240-259 138 138 138 254 254 254 F F F
13 260-279 138 138 138 256 256 256 F F F
14 280-299 136 136 136 251 251 251 F F F
15 300-319 136 136 136 252 252 252 F F F
16 320-339 139 139 139 258 258 258 F F F
17 340-359 132 132 132 243 243 243 F F F
18 360-379 138 138 138 255 255 255 F F F
19 380-399 138 138 138 255 255 255 F F F
20 400-419 136 136 136 249 249 249 F F F
21 420-439 136 136 136 249 249 249 F F F
22 440-459 139 139 139 258 258 258 F F F
23 460-479 135 135 135 248 248 248 F F F
24 480-499 138 138 138 256 256 256 F F F
25 500-519 131 131 131 242 242 242 F F F
26 520-539 136 136 136 252 252 252 F F F
27 540-559 137 137 137 253 253 253 F F F
28 560-579 139 139 139 258 258 258 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 139, Max flush: 258
Begin loop 70
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.924081058Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.970149001Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.971143395Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.971192345Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.971207975Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.971222496Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742}
{"msg":"Crucible 15902fd1-ec01-40e6-b3c1-1bb76fbc76dd has session id: 1eec314e-62d2-461b-97e8-9015eba0f855","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.9718136Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"Upstairs opts: Upstairs UUID: 15902fd1-ec01-40e6-b3c1-1bb76fbc76dd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.971877101Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"Crucible stats registered with UUID: 15902fd1-ec01-40e6-b3c1-1bb76fbc76dd","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.971903385Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:08.971916396Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.972519493Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"task":"crutest"}
{"msg":"The guest has requested activation with gen:140","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.972548345Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742}
{"msg":"15902fd1-ec01-40e6-b3c1-1bb76fbc76dd active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.972617672Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.972635451Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"0","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.972659796Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"0","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.972673168Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"1","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.972687089Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"1","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.9727031Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"2","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.972730133Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"2","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.972760324Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"io task","client":"2","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.972784779Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"io task","client":"0","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.972804797Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"io task","client":"1","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"ds_connection connected from Ok(127.0.0.1:40520)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.972953774Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"io task","client":"0","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"ds_connection connected from Ok(127.0.0.1:56654)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.972995159Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"io task","client":"1","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"ds_connection connected from Ok(127.0.0.1:45153)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.973101273Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"io task","client":"2","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.973765332Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"2","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.973804427Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"0","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.974445061Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"2","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.974605971Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"0","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.975089292Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.975157879Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.975388686Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"1","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.975897071Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"1","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"[0]R flush_numbers[0..12]: [240, 257, 247, 253, 257, 253, 253, 253, 253, 250, 246, 257]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976428872Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"downstairs","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"[0]R generation[0..12]: [130, 138, 134, 137, 138, 137, 137, 137, 137, 136, 134, 138]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976457535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"downstairs","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976475873Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"downstairs","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"[1]R flush_numbers[0..12]: [240, 257, 247, 253, 257, 253, 253, 253, 253, 250, 246, 257]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976498129Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"downstairs","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"[1]R generation[0..12]: [130, 138, 134, 137, 138, 137, 137, 137, 137, 136, 134, 138]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976515539Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"downstairs","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976532678Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"downstairs","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"[2]R flush_numbers[0..12]: [240, 257, 247, 253, 257, 253, 253, 253, 253, 250, 246, 257]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976549318Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"downstairs","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"[2]R generation[0..12]: [130, 138, 134, 137, 138, 137, 137, 137, 137, 136, 134, 138]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976565868Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"downstairs","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976582947Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"downstairs","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"Max found gen is 140","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:42:08.976598967Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"downstairs","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"Generation requested: 140 >= found:140","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976622273Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"downstairs","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"Next flush: 259","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976638023Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"downstairs","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976653873Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"":"downstairs","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976669643Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976684134Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"0","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976700274Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"1","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976716104Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"client":"2","":"downstairs","session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976732174Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976749344Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"15902fd1-ec01-40e6-b3c1-1bb76fbc76dd is now active with session: 1eec314e-62d2-461b-97e8-9015eba0f855","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976765074Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976779695Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742,"session_id":"1eec314e-62d2-461b-97e8-9015eba0f855"}
{"msg":"The guest has finished waiting for activation with:140","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:08.976794416Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 324 len 36864 data: 13 16 14 17 12 11 10 10 8
02/30 Flush
03/30 Write block 127 len 12288 data: 7 7 8
04/30 Read block 93 len 16384
05/30 Read block 73 len 4096
06/30 Read block 161 len 12288
07/30 Read block 484 len 32768
08/30 Read block 187 len 20480
09/30 Write block 69 len 24576 data: 13 16 17 12 14 13
10/30 Read block 391 len 32768
11/30 Write block 112 len 16384 data: 11 11 9 12
12/30 Write block 20 len 8192 data: 9 10
13/30 Write block 303 len 36864 data: 13 12 13 15 15 15 13 11 10
14/30 Write block 82 len 36864 data: 14 13 17 21 20 18 17 17 18
15/30 Flush
16/30 Write block 291 len 40960 data: 12 14 15 13 11 13 12 9 10 11
17/30 Flush
18/30 Write block 225 len 16384 data: 7 9 8 9
19/30 Read block 540 len 8192
20/30 Read block 189 len 20480
21/30 Write block 6 len 32768 data: 10 12 9 10 10 8 5 4
22/30 Read block 595 len 4096
23/30 Write block 39 len 12288 data: 8 10 12
24/30 Write block 58 len 32768 data: 11 10 10 10 13 14 14 16
25/30 Write block 95 len 36864 data: 15 13 12 14 12 14 15 16 17
26/30 Write block 160 len 20480 data: 8 13 11 9 13
27/30 Read block 492 len 24576
28/30 Write block 317 len 12288 data: 8 7 9
29/30 Write block 356 len 28672 data: 10 11 13 14 13 11 12
30/30 Read block 577 len 8192
----------------------------------------------------------------
Crucible gen:140 GIO:true work queues: Upstairs:0 downstairs:13
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1017 Acked Write 4 Done Done Done false
1018 Acked Read 2 Done Done Done false
1019 Acked Read 5 Done Done Done false
1020 Acked Write 8 Done Done Done false
1021 Acked Read 1 Done Done Done false
1022 Acked Write 3 Done Done Done false
1023 Acked Write 8 Done Done Done false
1024 Acked Write 9 Done Done Done false
1025 Acked Write 5 Done Done Done false
1026 Acked Read 6 Done Done Done false
1027 Acked Write 3 Done Done Done false
1028 Acked Write 7 Sent Done Sent false
1029 Acked Read 2 Sent Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 0 2 4
Done 11 13 11 35
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1016 1016 1016
Downstairs last five completed: 1016 1015 1014 1013 1012
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:08.998234739Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1742}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 130 130 130 240 240 240 T T T
1 020-039 140 140 140 260 260 260 T T T
2 040-059 134 134 134 247 247 247 T T T
3 060-079 140 140 140 260 260 260 T T T
4 080-099 140 140 140 260 260 260 T T T
5 100-119 140 140 140 260 260 260 T T T
6 120-139 140 140 140 260 260 260 F F F
7 140-159 137 137 137 253 253 253 F F F
8 160-179 137 137 137 253 253 253 T T T
9 180-199 136 136 136 250 250 250 F F F
10 200-219 134 134 134 246 246 246 F F F
11 220-239 138 138 138 257 257 257 T T T
12 240-259 138 138 138 254 254 254 F F F
13 260-279 138 138 138 256 256 256 F F F
14 280-299 140 140 140 261 261 261 F F F
15 300-319 140 140 140 261 261 261 T T T
16 320-339 140 140 140 259 259 259 F F F
17 340-359 132 132 132 243 243 243 T T T
18 360-379 138 138 138 255 255 255 T T T
19 380-399 138 138 138 255 255 255 F F F
20 400-419 136 136 136 249 249 249 F F F
21 420-439 136 136 136 249 249 249 F F F
22 440-459 139 139 139 258 258 258 F F F
23 460-479 135 135 135 248 248 248 F F F
24 480-499 138 138 138 256 256 256 F F F
25 500-519 131 131 131 242 242 242 F F F
26 520-539 136 136 136 252 252 252 F F F
27 540-559 137 137 137 253 253 253 F F F
28 560-579 139 139 139 258 258 258 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 140, Max flush: 261
A Difference in extent metadata is expected here
On loop 70
Verifying data now
No change after verify
Loop: 70 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 141 141 141 262 262 262 F F F
1 020-039 141 141 141 262 262 262 F F F
2 040-059 141 141 141 262 262 262 F F F
3 060-079 141 141 141 262 262 262 F F F
4 080-099 141 141 141 262 262 262 F F F
5 100-119 141 141 141 262 262 262 F F F
6 120-139 140 140 140 260 260 260 F F F
7 140-159 137 137 137 253 253 253 F F F
8 160-179 141 141 141 262 262 262 F F F
9 180-199 136 136 136 250 250 250 F F F
10 200-219 134 134 134 246 246 246 F F F
11 220-239 141 141 141 262 262 262 F F F
12 240-259 138 138 138 254 254 254 F F F
13 260-279 138 138 138 256 256 256 F F F
14 280-299 140 140 140 261 261 261 F F F
15 300-319 141 141 141 262 262 262 F F F
16 320-339 140 140 140 259 259 259 F F F
17 340-359 141 141 141 262 262 262 F F F
18 360-379 141 141 141 262 262 262 F F F
19 380-399 138 138 138 255 255 255 F F F
20 400-419 136 136 136 249 249 249 F F F
21 420-439 136 136 136 249 249 249 F F F
22 440-459 139 139 139 258 258 258 F F F
23 460-479 135 135 135 248 248 248 F F F
24 480-499 138 138 138 256 256 256 F F F
25 500-519 131 131 131 242 242 242 F F F
26 520-539 136 136 136 252 252 252 F F F
27 540-559 137 137 137 253 253 253 F F F
28 560-579 139 139 139 258 258 258 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 141, Max flush: 262
Begin loop 71
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.293008292Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:23.337380415Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.338123315Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.37031566Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.371286309Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.371323016Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.371336688Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.371350599Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750}
{"msg":"Crucible 7939d5f0-8863-48b5-a42c-0db3fa4bd76e has session id: 027fb14d-fd26-4e57-95d5-9dda840bbe66","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.371952386Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"Upstairs opts: Upstairs UUID: 7939d5f0-8863-48b5-a42c-0db3fa4bd76e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.371996729Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"Crucible stats registered with UUID: 7939d5f0-8863-48b5-a42c-0db3fa4bd76e","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.372023952Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:23.3720425Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.37246279Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"task":"crutest"}
{"msg":"The guest has requested activation with gen:142","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.372488774Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750}
{"msg":"7939d5f0-8863-48b5-a42c-0db3fa4bd76e active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.372547228Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.372573782Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"0","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.372603713Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"0","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.372618724Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"1","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.372633025Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"1","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.372645967Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"2","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.372659868Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"2","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.37267314Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"io task","client":"2","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.372686702Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"io task","client":"0","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.372700143Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"io task","client":"1","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"ds_connection connected from Ok(127.0.0.1:65167)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.372728456Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"io task","client":"2","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"ds_connection connected from Ok(127.0.0.1:61522)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.373043791Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"io task","client":"1","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49987)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.373087673Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"io task","client":"0","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.373397541Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"1","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.373425015Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"2","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.373958514Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"0","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.374114088Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"1","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.374139562Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"2","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.374603035Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"0","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.374718164Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.374748255Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"[0]R flush_numbers[0..12]: [262, 262, 262, 262, 262, 262, 260, 253, 262, 250, 246, 262]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375102645Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"downstairs","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"[0]R generation[0..12]: [141, 141, 141, 141, 141, 141, 140, 137, 141, 136, 134, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375129699Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"downstairs","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.37514444Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"downstairs","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"[1]R flush_numbers[0..12]: [262, 262, 262, 262, 262, 262, 260, 253, 262, 250, 246, 262]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375158431Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"downstairs","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"[1]R generation[0..12]: [141, 141, 141, 141, 141, 141, 140, 137, 141, 136, 134, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375171933Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"downstairs","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375185924Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"downstairs","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"[2]R flush_numbers[0..12]: [262, 262, 262, 262, 262, 262, 260, 253, 262, 250, 246, 262]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375199356Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"downstairs","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"[2]R generation[0..12]: [141, 141, 141, 141, 141, 141, 140, 137, 141, 136, 134, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375212747Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"downstairs","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375226309Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"downstairs","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
Wait for a query_work_queue command to finish before sending IO
{"msg":"Max found gen is 142","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375248075Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"downstairs","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"Generation requested: 142 >= found:142","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375261437Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"downstairs","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"Next flush: 263","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.37527391Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"downstairs","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375286482Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"":"downstairs","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375299274Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375310867Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"0","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375323499Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"1","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375336071Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"client":"2","":"downstairs","session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375348613Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375360396Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"7939d5f0-8863-48b5-a42c-0db3fa4bd76e is now active with session: 027fb14d-fd26-4e57-95d5-9dda840bbe66","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375372638Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375387529Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750,"session_id":"027fb14d-fd26-4e57-95d5-9dda840bbe66"}
{"msg":"The guest has finished waiting for activation with:142","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:23.375399492Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 182 len 8192 data: 17 16
02/30 Read block 523 len 8192
03/30 Flush
04/30 Write block 540 len 12288 data: 9 10 9
05/30 Write block 130 len 20480 data: 7 8 7 7 9
06/30 Flush
07/30 Write block 448 len 32768 data: 7 9 7 8 7 7 7 11
08/30 Read block 544 len 40960
09/30 Write block 215 len 20480 data: 10 8 8 9 7
10/30 Write block 308 len 8192 data: 16 14
11/30 Read block 275 len 4096
12/30 Flush
13/30 Write block 162 len 40960 data: 12 10 14 14 15 13 10 7 10 10
14/30 Read block 544 len 40960
15/30 Write block 133 len 28672 data: 8 10 9 9 10 11 12
16/30 Read block 457 len 36864
17/30 Read block 300 len 16384
18/30 Read block 90 len 40960
19/30 Read block 248 len 36864
20/30 Read block 559 len 16384
21/30 Read block 501 len 32768
22/30 Read block 446 len 32768
23/30 Read block 310 len 20480
24/30 Read block 488 len 4096
25/30 Read block 247 len 32768
26/30 Read block 452 len 16384
27/30 Read block 79 len 36864
28/30 Write block 201 len 12288 data: 7 9 8
29/30 Write block 280 len 24576 data: 15 14 12 8 7 8
30/30 Read block 41 len 32768
----------------------------------------------------------------
Crucible gen:142 GIO:true work queues: Upstairs:0 downstairs:18
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1012 Acked Write 10 Done Done Done false
1013 Acked Read 10 Done Done Done false
1014 Acked Write 7 Done Done Done false
1015 Acked Read 9 Done Done Done false
1016 Acked Read 4 Done Done Done false
1017 Acked Read 10 Done Done Done false
1018 Acked Read 9 Done Done Done false
1019 Acked Read 4 Done Done Done false
1020 Acked Read 8 Done Done Done false
1021 Acked Read 8 Done Done Done false
1022 Acked Read 5 Done Done Done false
1023 Acked Read 1 Done Done Done false
1024 Acked Read 8 Done Done Done false
1025 Acked Read 4 Done Done Done false
1026 Acked Read 9 Done Done Done false
1027 Acked Write 3 Done Done Done false
1028 Acked Write 6 Done Done Done false
1029 Acked Read 8 Sent Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 0 1 2
Done 17 18 17 52
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1011 1011 1011
Downstairs last five completed: 1011 1010 1009 1008 1007
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:23.395867687Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1750}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 141 141 141 262 262 262 F F F
1 020-039 141 141 141 262 262 262 F F F
2 040-059 141 141 141 262 262 262 F F F
3 060-079 141 141 141 262 262 262 F F F
4 080-099 141 141 141 262 262 262 F F F
5 100-119 141 141 141 262 262 262 F F F
6 120-139 142 142 142 264 264 264 T T T
7 140-159 137 137 137 253 253 253 F F F
8 160-179 141 141 141 262 262 262 T T T
9 180-199 142 142 142 263 263 263 F F F
10 200-219 142 142 142 265 265 265 T T T
11 220-239 141 141 141 262 262 262 F F F
12 240-259 138 138 138 254 254 254 F F F
13 260-279 138 138 138 256 256 256 F F F
14 280-299 140 140 140 261 261 261 T T T
15 300-319 142 142 142 265 265 265 F F F
16 320-339 140 140 140 259 259 259 F F F
17 340-359 141 141 141 262 262 262 F F F
18 360-379 141 141 141 262 262 262 F F F
19 380-399 138 138 138 255 255 255 F F F
20 400-419 136 136 136 249 249 249 F F F
21 420-439 136 136 136 249 249 249 F F F
22 440-459 142 142 142 265 265 265 F F F
23 460-479 135 135 135 248 248 248 F F F
24 480-499 138 138 138 256 256 256 F F F
25 500-519 131 131 131 242 242 242 F F F
26 520-539 136 136 136 252 252 252 F F F
27 540-559 142 142 142 264 264 264 F F F
28 560-579 139 139 139 258 258 258 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 142, Max flush: 265
A Difference in extent metadata is expected here
On loop 71
Verifying data now
No change after verify
Loop: 71 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 141 141 141 262 262 262 F F F
1 020-039 141 141 141 262 262 262 F F F
2 040-059 141 141 141 262 262 262 F F F
3 060-079 141 141 141 262 262 262 F F F
4 080-099 141 141 141 262 262 262 F F F
5 100-119 141 141 141 262 262 262 F F F
6 120-139 143 143 143 266 266 266 F F F
7 140-159 137 137 137 253 253 253 F F F
8 160-179 143 143 143 266 266 266 F F F
9 180-199 142 142 142 263 263 263 F F F
10 200-219 143 143 143 266 266 266 F F F
11 220-239 141 141 141 262 262 262 F F F
12 240-259 138 138 138 254 254 254 F F F
13 260-279 138 138 138 256 256 256 F F F
14 280-299 143 143 143 266 266 266 F F F
15 300-319 142 142 142 265 265 265 F F F
16 320-339 140 140 140 259 259 259 F F F
17 340-359 141 141 141 262 262 262 F F F
18 360-379 141 141 141 262 262 262 F F F
19 380-399 138 138 138 255 255 255 F F F
20 400-419 136 136 136 249 249 249 F F F
21 420-439 136 136 136 249 249 249 F F F
22 440-459 142 142 142 265 265 265 F F F
23 460-479 135 135 135 248 248 248 F F F
24 480-499 138 138 138 256 256 256 F F F
25 500-519 131 131 131 242 242 242 F F F
26 520-539 136 136 136 252 252 252 F F F
27 540-559 142 142 142 264 264 264 F F F
28 560-579 139 139 139 258 258 258 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 143, Max flush: 266
Begin loop 72
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.672816755Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.717642697Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.718504064Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.718549625Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.718564406Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.718579057Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758}
{"msg":"Crucible d021d5cf-70b0-4141-9ec7-553ae0ae82a3 has session id: 069ad29e-f1d7-4e5a-bb7b-b04366c97adb","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.719159358Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"Upstairs opts: Upstairs UUID: d021d5cf-70b0-4141-9ec7-553ae0ae82a3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.719209247Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"Crucible stats registered with UUID: d021d5cf-70b0-4141-9ec7-553ae0ae82a3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.719235261Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:27.719257037Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.719746244Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"task":"crutest"}
{"msg":"The guest has requested activation with gen:144","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.719783311Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758}
{"msg":"d021d5cf-70b0-4141-9ec7-553ae0ae82a3 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.719842455Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.719876634Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"0","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.719916849Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"0","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.719944212Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"1","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.719970496Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"1","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.719994391Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"2","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.720021234Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"2","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.720049307Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"io task","client":"2","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.720067186Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"io task","client":"1","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.720080827Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"io task","client":"0","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"ds_connection connected from Ok(127.0.0.1:44366)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.720283762Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"io task","client":"2","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"ds_connection connected from Ok(127.0.0.1:63914)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.720324117Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"io task","client":"0","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"ds_connection connected from Ok(127.0.0.1:35224)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.720345633Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"io task","client":"1","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.721090592Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"0","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.721128459Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"1","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.72115796Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"2","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.721812295Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"1","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.721853659Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"0","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.721890656Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"2","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722502677Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722527782Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"[0]R flush_numbers[0..12]: [262, 262, 262, 262, 262, 262, 266, 253, 266, 263, 266, 262]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.72259456Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"downstairs","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"[0]R generation[0..12]: [141, 141, 141, 141, 141, 141, 143, 137, 143, 142, 143, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722613459Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"downstairs","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.72262814Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"downstairs","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"[1]R flush_numbers[0..12]: [262, 262, 262, 262, 262, 262, 266, 253, 266, 263, 266, 262]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722645999Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"downstairs","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"[1]R generation[0..12]: [141, 141, 141, 141, 141, 141, 143, 137, 143, 142, 143, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.72265977Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"downstairs","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722673392Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"downstairs","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"[2]R flush_numbers[0..12]: [262, 262, 262, 262, 262, 262, 266, 253, 266, 263, 266, 262]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722686754Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"downstairs","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"[2]R generation[0..12]: [141, 141, 141, 141, 141, 141, 143, 137, 143, 142, 143, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722699785Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"downstairs","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722713707Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"downstairs","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"Max found gen is 144","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722726309Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"downstairs","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"Generation requested: 144 >= found:144","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722739081Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"downstairs","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"Next flush: 267","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:42:27.722751613Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"downstairs","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722767284Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"":"downstairs","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722779926Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722791549Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"0","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722804341Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"1","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722816873Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"client":"2","":"downstairs","session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722829425Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722843087Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"d021d5cf-70b0-4141-9ec7-553ae0ae82a3 is now active with session: 069ad29e-f1d7-4e5a-bb7b-b04366c97adb","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722855529Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722867122Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758,"session_id":"069ad29e-f1d7-4e5a-bb7b-b04366c97adb"}
{"msg":"The guest has finished waiting for activation with:144","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:27.722878895Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 265 len 28672 data: 13 11 8 9 12 11 13
02/30 Read block 514 len 24576
03/30 Read block 267 len 40960
04/30 Read block 200 len 28672
05/30 Read block 453 len 32768
06/30 Read block 70 len 12288
07/30 Write block 274 len 4096 data: 16
08/30 Read block 3 len 40960
09/30 Write block 372 len 4096 data: 12
10/30 Write block 214 len 24576 data: 9 11 9 9 10 8
11/30 Write block 294 len 24576 data: 14 12 14 13 10 11
12/30 Write block 187 len 28672 data: 11 7 6 7 11 10 12
13/30 Write block 477 len 12288 data: 12 14 11
14/30 Write block 563 len 20480 data: 8 9 9 8 8
15/30 Read block 176 len 40960
16/30 Read block 91 len 8192
17/30 Write block 105 len 4096 data: 18
18/30 Write block 280 len 24576 data: 16 15 13 9 8 9
19/30 Write block 172 len 28672 data: 9 8 9 14 13 12 13
20/30 Read block 187 len 8192
21/30 Read block 343 len 28672
22/30 Write block 327 len 16384 data: 18 13 12 11
23/30 Write block 476 len 12288 data: 11 13 15
24/30 Write block 383 len 16384 data: 7 7 7 10
25/30 Read block 554 len 36864
26/30 Read block 372 len 40960
27/30 Write block 504 len 20480 data: 6 7 8 8 9
28/30 Read block 231 len 36864
29/30 Write block 16 len 24576 data: 4 6 9 9 10 11
30/30 Read block 194 len 28672
----------------------------------------------------------------
Crucible gen:144 GIO:true work queues: Upstairs:0 downstairs:30
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Write 7 Done Done Done false
1001 Acked Read 6 Done Done Done false
1002 Acked Read 10 Done Done Done false
1003 Acked Read 7 Done Done Done false
1004 Acked Read 8 Done Done Done false
1005 Acked Read 3 Done Done Done false
1006 Acked Write 1 Done Done Done false
1007 Acked Read 10 Done Done Done false
1008 Acked Write 1 Done Done Done false
1009 Acked Write 6 Done Done Done false
1010 Acked Write 6 Done Done Done false
1011 Acked Write 7 Done Done Done false
1012 Acked Write 3 Done Done Done false
1013 Acked Write 5 Done Done Done false
1014 Acked Read 10 Done Done Done false
1015 Acked Read 2 Done Done Done false
1016 Acked Write 1 Done Done Done false
1017 Acked Write 6 Done Done Done false
1018 Acked Write 7 Done Done Done false
1019 Acked Read 2 Done Done Done false
1020 Acked Read 7 Done Done Done false
1021 Acked Write 4 Done Done Done false
1022 Acked Write 3 Done Done Done false
1023 Acked Write 4 Done Done Done false
1024 Acked Read 9 Done Done Done false
1025 Acked Read 10 Done Done Done false
1026 Acked Write 5 Done Done Done false
1027 Acked Read 9 Done Done Done false
1028 Acked Write 6 Done Done Done false
1029 Acked Read 7 Sent Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 0 1 2
Done 29 30 29 88
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 0 0 0
Downstairs last five completed:
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:27.742847279Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1758}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 141 141 141 262 262 262 T T T
1 020-039 141 141 141 262 262 262 T T T
2 040-059 141 141 141 262 262 262 F F F
3 060-079 141 141 141 262 262 262 F F F
4 080-099 141 141 141 262 262 262 F F F
5 100-119 141 141 141 262 262 262 T T T
6 120-139 143 143 143 266 266 266 F F F
7 140-159 137 137 137 253 253 253 F F F
8 160-179 143 143 143 266 266 266 T T T
9 180-199 142 142 142 263 263 263 T T T
10 200-219 143 143 143 266 266 266 T T T
11 220-239 141 141 141 262 262 262 F F F
12 240-259 138 138 138 254 254 254 F F F
13 260-279 138 138 138 256 256 256 T T T
14 280-299 143 143 143 266 266 266 T T T
15 300-319 142 142 142 265 265 265 F F F
16 320-339 140 140 140 259 259 259 T T T
17 340-359 141 141 141 262 262 262 F F F
18 360-379 141 141 141 262 262 262 T T T
19 380-399 138 138 138 255 255 255 T T T
20 400-419 136 136 136 249 249 249 F F F
21 420-439 136 136 136 249 249 249 F F F
22 440-459 142 142 142 265 265 265 F F F
23 460-479 135 135 135 248 248 248 T T T
24 480-499 138 138 138 256 256 256 F F F
25 500-519 131 131 131 242 242 242 T T T
26 520-539 136 136 136 252 252 252 F F F
27 540-559 142 142 142 264 264 264 F F F
28 560-579 139 139 139 258 258 258 T T T
29 580-599 133 133 133 245 245 245 F F F
Max gen: 143, Max flush: 266
A Difference in extent metadata is expected here
On loop 72
Verifying data now
No change after verify
Loop: 72 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 144 144 144 267 267 267 F F F
1 020-039 144 144 144 267 267 267 F F F
2 040-059 141 141 141 262 262 262 F F F
3 060-079 141 141 141 262 262 262 F F F
4 080-099 141 141 141 262 262 262 F F F
5 100-119 144 144 144 267 267 267 F F F
6 120-139 143 143 143 266 266 266 F F F
7 140-159 137 137 137 253 253 253 F F F
8 160-179 144 144 144 267 267 267 F F F
9 180-199 144 144 144 267 267 267 F F F
10 200-219 144 144 144 267 267 267 F F F
11 220-239 141 141 141 262 262 262 F F F
12 240-259 138 138 138 254 254 254 F F F
13 260-279 144 144 144 267 267 267 F F F
14 280-299 144 144 144 267 267 267 F F F
15 300-319 142 142 142 265 265 265 F F F
16 320-339 144 144 144 267 267 267 F F F
17 340-359 141 141 141 262 262 262 F F F
18 360-379 144 144 144 267 267 267 F F F
19 380-399 144 144 144 267 267 267 F F F
20 400-419 136 136 136 249 249 249 F F F
21 420-439 136 136 136 249 249 249 F F F
22 440-459 142 142 142 265 265 265 F F F
23 460-479 144 144 144 267 267 267 F F F
24 480-499 138 138 138 256 256 256 F F F
25 500-519 144 144 144 267 267 267 F F F
26 520-539 136 136 136 252 252 252 F F F
27 540-559 142 142 142 264 264 264 F F F
28 560-579 144 144 144 267 267 267 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 144, Max flush: 267
Begin loop 73
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.052935479Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:42.10339891Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.103976182Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.136055677Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.136706994Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.136741872Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.136755834Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.136770105Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766}
{"msg":"Crucible 1f14dcd1-c6ea-41b8-b93e-e55d74755040 has session id: 75cc0d4e-f1ce-450a-a151-6af6932ac8ff","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.137356812Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"Upstairs opts: Upstairs UUID: 1f14dcd1-c6ea-41b8-b93e-e55d74755040, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.137398985Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"Crucible stats registered with UUID: 1f14dcd1-c6ea-41b8-b93e-e55d74755040","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.13742393Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:42.137444297Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.137818825Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"task":"crutest"}
{"msg":"The guest has requested activation with gen:146","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.137851715Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766}
{"msg":"1f14dcd1-c6ea-41b8-b93e-e55d74755040 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.137974389Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.138014864Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"0","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.138041637Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"0","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.138055939Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"1","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.13807047Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"1","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.138083582Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"2","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.138100971Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"2","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.138123197Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"io task","client":"2","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.138149361Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"io task","client":"0","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.138174535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"io task","client":"1","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47575)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.138201289Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"io task","client":"2","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"ds_connection connected from Ok(127.0.0.1:44293)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.138227342Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"io task","client":"1","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"ds_connection connected from Ok(127.0.0.1:58142)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.138253576Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"io task","client":"0","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.13889336Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"2","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.139041878Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"1","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.139069901Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"0","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.139598793Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"1","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.139641157Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"2","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.139713472Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"0","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140165962Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140264311Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"[0]R flush_numbers[0..12]: [267, 267, 262, 262, 262, 267, 266, 253, 267, 267, 267, 262]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140354835Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"downstairs","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"[0]R generation[0..12]: [144, 144, 141, 141, 141, 144, 143, 137, 144, 144, 144, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140384677Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"downstairs","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140400776Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"downstairs","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"[1]R flush_numbers[0..12]: [267, 267, 262, 262, 262, 267, 266, 253, 267, 267, 267, 262]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140415048Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"downstairs","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"[1]R generation[0..12]: [144, 144, 141, 141, 141, 144, 143, 137, 144, 144, 144, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140428989Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"downstairs","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.14044275Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"downstairs","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"[2]R flush_numbers[0..12]: [267, 267, 262, 262, 262, 267, 266, 253, 267, 267, 267, 262]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.14046012Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"downstairs","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"[2]R generation[0..12]: [144, 144, 141, 141, 141, 144, 143, 137, 144, 144, 144, 141]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140474091Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"downstairs","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140487843Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"downstairs","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"Max found gen is 145","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140505462Z","hostname"Wait for a query_work_queue command to finish before sending IO
:"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"downstairs","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"Generation requested: 146 >= found:145","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140530176Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"downstairs","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"Next flush: 268","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140543098Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"downstairs","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.14055578Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"":"downstairs","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140568613Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140580565Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"0","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140593827Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"1","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140606719Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"client":"2","":"downstairs","session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140619571Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140631624Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"1f14dcd1-c6ea-41b8-b93e-e55d74755040 is now active with session: 75cc0d4e-f1ce-450a-a151-6af6932ac8ff","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140644156Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.140655789Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766,"session_id":"75cc0d4e-f1ce-450a-a151-6af6932ac8ff"}
{"msg":"The guest has finished waiting for activation with:146","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:42.14067003Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 157 len 40960 data: 12 13 12 9 14 13 11 15 15 16
02/30 Write block 515 len 24576 data: 9 9 8 9 10 10
03/30 Flush
04/30 Write block 366 len 36864 data: 8 8 12 13 14 14 13 15 16
05/30 Write block 229 len 24576 data: 9 13 11 8 8 8
06/30 Flush
07/30 Write block 486 len 16384 data: 10 13 13 10
08/30 Read block 70 len 36864
09/30 Write block 437 len 28672 data: 9 8 8 10 10 10 9
10/30 Write block 122 len 40960 data: 12 8 8 5 6 8 8 9 8 9
11/30 Read block 525 len 12288
12/30 Write block 138 len 36864 data: 12 13 11 12 13 13 12 13 14
13/30 Read block 509 len 24576
14/30 Write block 140 len 12288 data: 12 13 14
15/30 Read block 539 len 8192
16/30 Write block 281 len 36864 data: 16 14 10 9 10 7 7 7 9
17/30 Write block 558 len 16384 data: 12 13 12 9
18/30 Read block 417 len 24576
19/30 Read block 441 len 40960
20/30 Write block 490 len 28672 data: 12 14 16 15 11 16 17
21/30 Read block 94 len 32768
22/30 Write block 272 len 20480 data: 12 12 17 18 16
23/30 Flush
24/30 Write block 360 len 4096 data: 14
25/30 Write block 464 len 8192 data: 12 11
26/30 Read block 334 len 32768
27/30 Write block 473 len 36864 data: 10 10 10 12 14 16 12 13 12
28/30 Write block 414 len 20480 data: 7 6 10 12 13
29/30 Read block 201 len 16384
30/30 Read block 203 len 32768
----------------------------------------------------------------
Crucible gen:146 GIO:true work queues: Upstairs:0 downstairs:7
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1023 Acked Write 1 Done Done Done false
1024 Acked Write 2 Done Done Done false
1025 Acked Read 8 Done Done Done false
1026 Acked Write 9 Done Done Done false
1027 Acked Write 5 Done Done Done false
1028 Acked Read 4 Done Done Done false
1029 Acked Read 8 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 1 2
Done 7 6 6 19
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1022 1022 1022
Downstairs last five completed: 1022 1021 1020 1019 1018
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:42.162266456Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1766}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 144 144 144 267 267 267 F F F
1 020-039 144 144 144 267 267 267 F F F
2 040-059 141 141 141 262 262 262 F F F
3 060-079 141 141 141 262 262 262 F F F
4 080-099 141 141 141 262 262 262 F F F
5 100-119 144 144 144 267 267 267 F F F
6 120-139 146 146 146 270 270 270 F F F
7 140-159 146 146 146 270 270 270 F F F
8 160-179 146 146 146 268 268 268 F F F
9 180-199 144 144 144 267 267 267 F F F
10 200-219 144 144 144 267 267 267 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 138 138 138 254 254 254 F F F
13 260-279 146 146 146 270 270 270 F F F
14 280-299 146 146 146 270 270 270 F F F
15 300-319 142 142 142 265 265 265 F F F
16 320-339 144 144 144 267 267 267 F F F
17 340-359 141 141 141 262 262 262 F F F
18 360-379 146 146 146 269 269 269 T T T
19 380-399 144 144 144 267 267 267 F F F
20 400-419 136 136 136 249 249 249 T T T
21 420-439 146 146 146 270 270 270 F F F
22 440-459 146 146 146 270 270 270 F F F
23 460-479 144 144 144 267 267 267 T T T
24 480-499 146 146 146 270 270 270 T T T
25 500-519 146 146 146 268 268 268 F F F
26 520-539 146 146 146 268 268 268 F F F
27 540-559 146 146 146 270 270 270 F F F
28 560-579 146 146 146 270 270 270 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 146, Max flush: 270
A Difference in extent metadata is expected here
On loop 73
Verifying data now
No change after verify
Loop: 73 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 144 144 144 267 267 267 F F F
1 020-039 144 144 144 267 267 267 F F F
2 040-059 141 141 141 262 262 262 F F F
3 060-079 141 141 141 262 262 262 F F F
4 080-099 141 141 141 262 262 262 F F F
5 100-119 144 144 144 267 267 267 F F F
6 120-139 146 146 146 270 270 270 F F F
7 140-159 146 146 146 270 270 270 F F F
8 160-179 146 146 146 268 268 268 F F F
9 180-199 144 144 144 267 267 267 F F F
10 200-219 144 144 144 267 267 267 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 138 138 138 254 254 254 F F F
13 260-279 146 146 146 270 270 270 F F F
14 280-299 146 146 146 270 270 270 F F F
15 300-319 142 142 142 265 265 265 F F F
16 320-339 144 144 144 267 267 267 F F F
17 340-359 141 141 141 262 262 262 F F F
18 360-379 147 147 147 271 271 271 F F F
19 380-399 144 144 144 267 267 267 F F F
20 400-419 147 147 147 271 271 271 F F F
21 420-439 146 146 146 270 270 270 F F F
22 440-459 146 146 146 270 270 270 F F F
23 460-479 147 147 147 271 271 271 F F F
24 480-499 147 147 147 271 271 271 F F F
25 500-519 146 146 146 268 268 268 F F F
26 520-539 146 146 146 268 268 268 F F F
27 540-559 146 146 146 270 270 270 F F F
28 560-579 146 146 146 270 270 270 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 147, Max flush: 271
Begin loop 74
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.439043599Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.485846837Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.486882855Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.486930186Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.486945606Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.486964954Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774}
{"msg":"Crucible fb5061d4-aa75-4c5c-9ca6-6d0c0b4d6aa8 has session id: 10251f62-7e68-414d-8d66-6d0ce9915f31","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.487631471Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"Upstairs opts: Upstairs UUID: fb5061d4-aa75-4c5c-9ca6-6d0c0b4d6aa8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.48766592Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"Crucible stats registered with UUID: fb5061d4-aa75-4c5c-9ca6-6d0c0b4d6aa8","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.487684229Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:46.487696791Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.488142285Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"task":"crutest"}
{"msg":"The guest has requested activation with gen:148","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.488168179Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774}
{"msg":"fb5061d4-aa75-4c5c-9ca6-6d0c0b4d6aa8 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.488232859Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.488249828Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"0","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.488273124Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"0","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.488286566Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"1","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.488300557Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"1","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.488356402Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"2","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.48840963Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"2","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.488433355Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"io task","client":"2","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.48845778Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"io task","client":"0","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.488484633Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"io task","client":"1","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57757)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.488509518Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"io task","client":"1","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"ds_connection connected from Ok(127.0.0.1:44779)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.488765739Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"io task","client":"2","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39834)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.488809992Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"io task","client":"0","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.489428629Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"2","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.489481136Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"0","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.489496786Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"1","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.490170699Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"2","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.490199341Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"0","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.490221088Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"1","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.490810333Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.490845301Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"[0]R flush_numbers[0..12]: [267, 267, 262, 262, 262, 267, 270, 270, 268, 267, 267, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.490902146Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"downstairs","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"[0]R generation[0..12]: [144, 144, 141, 141, 141, 144, 146, 146, 146, 144, 144, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.490934066Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"downstairs","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.490950296Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"downstairs","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"[1]R flush_numbers[0..12]: [267, 267, 262, 262, 262, 267, 270, 270, 268, 267, 267, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.490964478Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"downstairs","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"[1]R generation[0..12]: [144, 144, 141, 141, 141, 144, 146, 146, 146, 144, 144, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.490978259Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"downstairs","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.490994599Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"downstairs","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"[2]R flush_numbers[0..12]: [267, 267, 262, 262, 262, 267, 270, 270, 268, 267, 267, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491018444Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"downstairs","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"[2]R generation[0..12]: [144, 144, 141, 141, 141, 144, 146, 146, 146, 144, 144, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491033415Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"downstairs","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:42:46.491047476Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"downstairs","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"Max found gen is 148","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491064166Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"downstairs","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"Generation requested: 148 >= found:148","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491087012Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"downstairs","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"Next flush: 272","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.49110625Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"downstairs","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491119402Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"":"downstairs","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491132144Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491143647Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"0","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491173338Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"1","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491200112Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"client":"2","":"downstairs","session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491215602Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491227824Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"fb5061d4-aa75-4c5c-9ca6-6d0c0b4d6aa8 is now active with session: 10251f62-7e68-414d-8d66-6d0ce9915f31","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491240197Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491251829Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774,"session_id":"10251f62-7e68-414d-8d66-6d0ce9915f31"}
{"msg":"The guest has finished waiting for activation with:148","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:42:46.491270738Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 388 len 4096 data: 11
02/30 Write block 240 len 36864 data: 6 9 12 14 13 12 15 16 15
03/30 Read block 50 len 36864
04/30 Write block 412 len 8192 data: 8 8
05/30 Read block 387 len 24576
06/30 Read block 26 len 40960
07/30 Read block 116 len 8192
08/30 Write block 404 len 40960 data: 12 13 11 11 10 10 10 10 9 9
09/30 Read block 79 len 4096
10/30 Write block 117 len 16384 data: 18 15 17 14
11/30 Read block 274 len 20480
12/30 Read block 213 len 24576
13/30 Write block 182 len 24576 data: 18 17 11 11 10 12
14/30 Read block 366 len 24576
15/30 Write block 492 len 16384 data: 17 16 12 17
16/30 Write block 72 len 4096 data: 13
17/30 Read block 592 len 8192
18/30 Read block 588 len 24576
19/30 Flush
20/30 Write block 447 len 36864 data: 8 8 10 8 9 8 8 8 12
21/30 Write block 120 len 40960 data: 15 14 13 9 9 6 7 9 9 10
22/30 Write block 555 len 8192 data: 11 14
23/30 Read block 263 len 20480
24/30 Read block 341 len 16384
25/30 Write block 529 len 28672 data: 16 15 14 14 14 13 15
26/30 Write block 71 len 36864 data: 18 14 15 14 12 9 7 9 8
27/30 Write block 394 len 36864 data: 11 11 10 9 8 8 11 12 14
28/30 Write block 511 len 8192 data: 10 9
29/30 Write block 480 len 36864 data: 14 13 12 12 12 11 11 14 14
30/30 Read block 186 len 36864
----------------------------------------------------------------
Crucible gen:148 GIO:true work queues: Upstairs:0 downstairs:11
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1019 Acked Write 9 Done Done Done false
1020 Acked Write 10 Done Done Done false
1021 Acked Write 2 Done Done Done false
1022 Acked Read 5 Done Done Done false
1023 Acked Read 4 Done Done Done false
1024 Acked Write 7 Done Done Done false
1025 Acked Write 9 Done Done Done false
1026 Acked Write 9 Done Done Done false
1027 Acked Write 2 Done Done Done false
1028 Acked Write 9 Done Done Sent false
1029 Acked Read 9 Sent Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 0 2 3
Done 10 11 9 30
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1018 1018 1018
Downstairs last five completed: 1018 1017 1016 1015 1014
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:42:46.511713369Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1774}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 144 144 144 267 267 267 F F F
1 020-039 144 144 144 267 267 267 F F F
2 040-059 141 141 141 262 262 262 F F F
3 060-079 148 148 148 272 272 272 T T T
4 080-099 141 141 141 262 262 262 F F F
5 100-119 148 148 148 272 272 272 F F F
6 120-139 148 148 148 272 272 272 T T T
7 140-159 146 146 146 270 270 270 F F F
8 160-179 146 146 146 268 268 268 F F F
9 180-199 148 148 148 272 272 272 F F F
10 200-219 144 144 144 267 267 267 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 148 148 148 272 272 272 F F F
13 260-279 146 146 146 270 270 270 F F F
14 280-299 146 146 146 270 270 270 F F F
15 300-319 142 142 142 265 265 265 F F F
16 320-339 144 144 144 267 267 267 F F F
17 340-359 141 141 141 262 262 262 F F F
18 360-379 147 147 147 271 271 271 F F F
19 380-399 148 148 148 272 272 272 T T T
20 400-419 148 148 148 272 272 272 T T T
21 420-439 146 146 146 270 270 270 F F F
22 440-459 146 146 146 270 270 270 T T T
23 460-479 147 147 147 271 271 271 F F F
24 480-499 148 148 148 272 272 272 T T T
25 500-519 146 146 146 268 268 268 T T T
26 520-539 146 146 146 268 268 268 T T T
27 540-559 146 146 146 270 270 270 T T T
28 560-579 146 146 146 270 270 270 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 148, Max flush: 272
A Difference in extent metadata is expected here
On loop 74
Verifying data now
No change after verify
Loop: 74 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 144 144 144 267 267 267 F F F
1 020-039 144 144 144 267 267 267 F F F
2 040-059 141 141 141 262 262 262 F F F
3 060-079 149 149 149 273 273 273 F F F
4 080-099 141 141 141 262 262 262 F F F
5 100-119 148 148 148 272 272 272 F F F
6 120-139 149 149 149 273 273 273 F F F
7 140-159 146 146 146 270 270 270 F F F
8 160-179 146 146 146 268 268 268 F F F
9 180-199 148 148 148 272 272 272 F F F
10 200-219 144 144 144 267 267 267 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 148 148 148 272 272 272 F F F
13 260-279 146 146 146 270 270 270 F F F
14 280-299 146 146 146 270 270 270 F F F
15 300-319 142 142 142 265 265 265 F F F
16 320-339 144 144 144 267 267 267 F F F
17 340-359 141 141 141 262 262 262 F F F
18 360-379 147 147 147 271 271 271 F F F
19 380-399 149 149 149 273 273 273 F F F
20 400-419 149 149 149 273 273 273 F F F
21 420-439 146 146 146 270 270 270 F F F
22 440-459 149 149 149 273 273 273 F F F
23 460-479 147 147 147 271 271 271 F F F
24 480-499 149 149 149 273 273 273 F F F
25 500-519 149 149 149 273 273 273 F F F
26 520-539 149 149 149 273 273 273 F F F
27 540-559 149 149 149 273 273 273 F F F
28 560-579 146 146 146 270 270 270 F F F
29 580-599 133 133 133 245 245 245 F F F
Max gen: 149, Max flush: 273
Begin loop 75
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.794262091Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.840256039Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.841324408Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.841382102Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.841405847Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.841428253Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782}
{"msg":"Crucible 5583c51a-3a52-4a08-8059-5a6332ccd0af has session id: 1e12a4ff-12cb-4255-8ff3-03c2a90e3a51","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.8420636Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"Upstairs opts: Upstairs UUID: 5583c51a-3a52-4a08-8059-5a6332ccd0af, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.842114109Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"Crucible stats registered with UUID: 5583c51a-3a52-4a08-8059-5a6332ccd0af","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.842133317Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:00.842146988Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.842508505Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"task":"crutest"}
{"msg":"The guest has requested activation with gen:150","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.842565539Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782}
{"msg":"5583c51a-3a52-4a08-8059-5a6332ccd0af active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.842589035Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.842607044Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"0","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.84262875Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"0","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.84264468Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"1","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.842658801Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"1","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.842671504Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"2","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.842684955Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"2","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.842697657Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"io task","client":"2","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.842710739Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"io task","client":"0","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.842728018Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"io task","client":"1","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"ds_connection connected from Ok(127.0.0.1:51484)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.842758579Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"io task","client":"2","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"ds_connection connected from Ok(127.0.0.1:34313)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.843144001Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"io task","client":"0","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"ds_connection connected from Ok(127.0.0.1:37667)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.843180408Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"io task","client":"1","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.843541874Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"0","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.843583988Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"1","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.84379202Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"2","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.84435654Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"1","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.844395836Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"0","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.844445275Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"2","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.844971469Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845095463Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"[0]R flush_numbers[0..12]: [267, 267, 262, 273, 262, 272, 273, 270, 268, 272, 267, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845151857Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"downstairs","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"[0]R generation[0..12]: [144, 144, 141, 149, 141, 148, 149, 146, 146, 148, 144, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845170336Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"downstairs","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845184497Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"downstairs","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"[1]R flush_numbers[0..12]: [267, 267, 262, 273, 262, 272, 273, 270, 268, 272, 267, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845198379Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"downstairs","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"[1]R generation[0..12]: [144, 144, 141, 149, 141, 148, 149, 146, 146, 148, 144, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.84521178Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"downstairs","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845225462Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"downstairs","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"[2]R flush_numbers[0..12]: [267, 267, 262, 273, 262, 272, 273, 270, 268, 272, 267, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845238814Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"downstairs","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"[2]R generation[0..12]: [144, 144, 141, 149, 141, 148, 149, 146, 146, 148, 144, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845252325Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"downstairs","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845265817Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"downstairs","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"Max found gen is 150","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845282567Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"downstairs","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"Generation requested: 150 >= found:150","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845295779Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"downstairs","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}Wait for a query_work_queue command to finish before sending IO
{"msg":"Next flush: 274","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845314837Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"downstairs","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845327609Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"":"downstairs","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845340431Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845351924Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"0","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845364746Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"1","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845377198Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"client":"2","":"downstairs","session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.84539003Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845401973Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"5583c51a-3a52-4a08-8059-5a6332ccd0af is now active with session: 1e12a4ff-12cb-4255-8ff3-03c2a90e3a51","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845414335Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.845425918Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782,"session_id":"1e12a4ff-12cb-4255-8ff3-03c2a90e3a51"}
{"msg":"The guest has finished waiting for activation with:150","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:00.84543774Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 492 len 12288 data: 18 17 13
02/30 Flush
03/30 Write block 142 len 28672 data: 15 14 13 14 15 14 14
04/30 Read block 386 len 40960
05/30 Read block 56 len 36864
06/30 Write block 413 len 32768 data: 10 8 7 11 13 14 14 14
07/30 Write block 533 len 28672 data: 15 14 16 12 11 9 10
08/30 Write block 136 len 20480 data: 10 11 13 14 13
09/30 Read block 182 len 24576
10/30 Read block 452 len 36864
11/30 Write block 203 len 16384 data: 9 11 11 11
12/30 Read block 181 len 20480
13/30 Flush
14/30 Write block 589 len 8192 data: 12 11
15/30 Read block 17 len 32768
16/30 Write block 58 len 28672 data: 12 11 11 11 14 15 15
17/30 Write block 338 len 32768 data: 10 10 10 12 14 14 16 14
18/30 Write block 336 len 32768 data: 7 8 11 11 11 13 15 15
19/30 Read block 103 len 12288
20/30 Read block 273 len 20480
21/30 Read block 379 len 40960
22/30 Flush
23/30 Write block 4 len 28672 data: 9 10 11 13 10 11 11
24/30 Read block 0 len 24576
25/30 Write block 323 len 24576 data: 13 14 17 15 19 14
26/30 Read block 118 len 16384
27/30 Write block 393 len 32768 data: 14 12 12 11 10 9 9 12
28/30 Write block 373 len 20480 data: 16 17 16 17 16
29/30 Write block 48 len 24576 data: 7 8 7 8 9 9
30/30 Write block 167 len 4096 data: 14
----------------------------------------------------------------
Crucible gen:150 GIO:true work queues: Upstairs:0 downstairs:8
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1022 Acked Write 7 Done Done Done false
1023 Acked Read 6 Done Done Done false
1024 Acked Write 6 Done Done Done false
1025 Acked Read 4 Done Done Done false
1026 Acked Write 8 Sent Sent Sent false
1027 Acked Write 5 Sent Sent Sent false
1028 Acked Write 6 Sent Sent Sent false
1029 Acked Write 1 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 4 4 4 12
Done 4 4 4 12
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1021 1021 1021
Downstairs last five completed: 1021 1020 1019 1018 1017
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:00.866147026Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1782}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 144 144 144 267 267 267 T T T
1 020-039 144 144 144 267 267 267 F F F
2 040-059 150 150 150 276 276 276 T T T
3 060-079 150 150 150 276 276 276 F F F
4 080-099 141 141 141 262 262 262 F F F
5 100-119 148 148 148 272 272 272 F F F
6 120-139 150 150 150 275 275 275 F F F
7 140-159 150 150 150 275 275 275 F F F
8 160-179 146 146 146 268 268 268 T T T
9 180-199 148 148 148 272 272 272 F F F
10 200-219 150 150 150 275 275 275 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 148 148 148 272 272 272 F F F
13 260-279 146 146 146 270 270 270 F F F
14 280-299 146 146 146 270 270 270 F F F
15 300-319 142 142 142 265 265 265 F F F
16 320-339 150 150 150 276 276 276 T T T
17 340-359 150 150 150 276 276 276 F F F
18 360-379 147 147 147 271 271 271 T T T
19 380-399 149 149 149 273 273 273 T T T
20 400-419 150 150 150 275 275 275 T T T
21 420-439 150 150 150 275 275 275 F F F
22 440-459 149 149 149 273 273 273 F F F
23 460-479 147 147 147 271 271 271 F F F
24 480-499 150 150 150 274 274 274 F F F
25 500-519 149 149 149 273 273 273 F F F
26 520-539 150 150 150 275 275 275 F F F
27 540-559 149 149 149 273 273 273 F F F
28 560-579 146 146 146 270 270 270 F F F
29 580-599 150 150 150 276 276 276 F F F
Max gen: 150, Max flush: 276
A Difference in extent metadata is expected here
On loop 75
Verifying data now
No change after verify
Loop: 75 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 151 151 151 277 277 277 F F F
1 020-039 144 144 144 267 267 267 F F F
2 040-059 151 151 151 277 277 277 F F F
3 060-079 150 150 150 276 276 276 F F F
4 080-099 141 141 141 262 262 262 F F F
5 100-119 148 148 148 272 272 272 F F F
6 120-139 150 150 150 275 275 275 F F F
7 140-159 150 150 150 275 275 275 F F F
8 160-179 151 151 151 277 277 277 F F F
9 180-199 148 148 148 272 272 272 F F F
10 200-219 150 150 150 275 275 275 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 148 148 148 272 272 272 F F F
13 260-279 146 146 146 270 270 270 F F F
14 280-299 146 146 146 270 270 270 F F F
15 300-319 142 142 142 265 265 265 F F F
16 320-339 151 151 151 277 277 277 F F F
17 340-359 150 150 150 276 276 276 F F F
18 360-379 151 151 151 277 277 277 F F F
19 380-399 151 151 151 277 277 277 F F F
20 400-419 151 151 151 277 277 277 F F F
21 420-439 150 150 150 275 275 275 F F F
22 440-459 149 149 149 273 273 273 F F F
23 460-479 147 147 147 271 271 271 F F F
24 480-499 150 150 150 274 274 274 F F F
25 500-519 149 149 149 273 273 273 F F F
26 520-539 150 150 150 275 275 275 F F F
27 540-559 149 149 149 273 273 273 F F F
28 560-579 146 146 146 270 270 270 F F F
29 580-599 150 150 150 276 276 276 F F F
Max gen: 151, Max flush: 277
Begin loop 76
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.138689153Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.184787677Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.185817959Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.185861442Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.185877902Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.185891963Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790}
{"msg":"Crucible 53f38877-43c6-4af7-a1c4-30473dffc200 has session id: f9a5364d-f011-445e-8d9f-2dd1853d8535","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.186487834Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"Upstairs opts: Upstairs UUID: 53f38877-43c6-4af7-a1c4-30473dffc200, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.186588612Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"Crucible stats registered with UUID: 53f38877-43c6-4af7-a1c4-30473dffc200","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.186617454Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:05.186632205Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.187080787Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"task":"crutest"}
{"msg":"The guest has requested activation with gen:152","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.187104962Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790}
{"msg":"53f38877-43c6-4af7-a1c4-30473dffc200 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.187193198Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.187216253Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"0","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.187241897Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"0","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.187255649Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"1","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.18726931Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"1","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.187300961Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"2","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.187315422Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"2","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.187328174Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"io task","client":"2","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.187341515Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.187362682Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"ds_connection connected from Ok(127.0.0.1:65013)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.18768956Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"io task","client":"2","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"ds_connection connected from Ok(127.0.0.1:65023)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.187740568Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"io task","client":"0","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"ds_connection connected from Ok(127.0.0.1:35403)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.18777031Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"io task","client":"1","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.188251852Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"0","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.188289708Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"1","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.188494601Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"2","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.188874137Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"1","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.188931131Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"0","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189144439Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"2","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189576242Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189612009Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"[0]R flush_numbers[0..12]: [277, 267, 277, 276, 262, 272, 275, 275, 277, 272, 275, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189735353Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"downstairs","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"[0]R generation[0..12]: [151, 144, 151, 150, 141, 148, 150, 150, 151, 148, 150, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189754621Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"downstairs","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189768903Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"downstairs","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"[1]R flush_numbers[0..12]: [277, 267, 277, 276, 262, 272, 275, 275, 277, 272, 275, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189783154Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"downstairs","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"[1]R generation[0..12]: [151, 144, 151, 150, 141, 148, 150, 150, 151, 148, 150, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189801103Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"downstairs","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189815234Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"downstairs","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"[2]R flush_numbers[0..12]: [277, 267, 277, 276, 262, 272, 275, 275, 277, 272, 275, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189828546Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"downstairs","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"[2]R generation[0..12]: [151, 144, 151, 150, 141, 148, 150, 150, 151, 148, 150, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189841937Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"downstairs","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189855429Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"downstairs","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"Max found gen is 152","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189867981Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"downstairs","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msgWait for a query_work_queue command to finish before sending IO
":"Generation requested: 152 >= found:152","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189887059Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"downstairs","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"Next flush: 278","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189899951Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"downstairs","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189912574Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"":"downstairs","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189925335Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189937048Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"0","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.18994986Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"1","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189962623Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"client":"2","":"downstairs","session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189975464Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.189989336Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"53f38877-43c6-4af7-a1c4-30473dffc200 is now active with session: f9a5364d-f011-445e-8d9f-2dd1853d8535","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.190001938Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.190013651Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790,"session_id":"f9a5364d-f011-445e-8d9f-2dd1853d8535"}
{"msg":"The guest has finished waiting for activation with:152","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:05.190025863Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 264 len 40960 data: 13 14 12 9 10 13 12 14 13 13
02/30 Read block 171 len 8192
03/30 Write block 326 len 24576 data: 16 20 15 13 12 11
04/30 Read block 377 len 16384
05/30 Flush
06/30 Write block 86 len 36864 data: 21 19 18 18 19 15 18 20 15
07/30 Read block 194 len 4096
08/30 Read block 175 len 12288
09/30 Read block 411 len 32768
10/30 Read block 495 len 32768
11/30 Write block 164 len 28672 data: 16 16 17 15 11 8 11
12/30 Write block 590 len 28672 data: 12 10 10 11 9 9 11
13/30 Read block 560 len 4096
14/30 Write block 534 len 36864 data: 15 17 13 12 10 11 10 11 10
15/30 Read block 144 len 40960
16/30 Read block 140 len 32768
17/30 Read block 343 len 40960
18/30 Write block 391 len 40960 data: 14 14 15 13 13 12 11 10 10 13
19/30 Read block 456 len 4096
20/30 Write block 347 len 32768 data: 12 16 18 15 14 12 11 10
21/30 Write block 535 len 24576 data: 18 14 13 11 12 11
22/30 Read block 9 len 28672
23/30 Read block 335 len 4096
24/30 Read block 2 len 16384
25/30 Write block 49 len 12288 data: 9 8 9
26/30 Read block 578 len 32768
27/30 Read block 536 len 4096
28/30 Write block 58 len 4096 data: 13
29/30 Read block 599 len 4096
30/30 Write block 182 len 16384 data: 19 18 12 12
----------------------------------------------------------------
Crucible gen:152 GIO:true work queues: Upstairs:0 downstairs:25
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1005 Acked Write 9 Done Done Done false
1006 Acked Read 1 Done Done Done false
1007 Acked Read 3 Done Done Done false
1008 Acked Read 8 Done Done Done false
1009 Acked Read 8 Done Done Done false
1010 Acked Write 7 Done Done Done false
1011 Acked Write 7 Done Done Done false
1012 Acked Read 1 Done Done Done false
1013 Acked Write 9 Done Done Done false
1014 Acked Read 10 Done Done Done false
1015 Acked Read 8 Done Done Done false
1016 Acked Read 10 Done Done Done false
1017 Acked Write 10 Done Done Done false
1018 Acked Read 1 Done Done Done false
1019 Acked Write 8 Done Done Done false
1020 Acked Write 6 Done Done Done false
1021 Acked Read 7 Done Done Done false
1022 Acked Read 1 Done Done Done false
1023 Acked Read 4 Done Done Done false
1024 Acked Write 3 Done Done Done false
1025 Acked Read 8 Done Done Done false
1026 Acked Read 1 Done Done Done false
1027 Acked Write 1 Done Done Done false
1028 Acked Read 1 Done Done Done false
1029 Acked Write 4 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 24 24 24 72
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1004 1004 1004
Downstairs last five completed: 1004 1003 1002 1001 1000
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:05.210865988Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1790}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 151 151 151 277 277 277 F F F
1 020-039 144 144 144 267 267 267 F F F
2 040-059 151 151 151 277 277 277 T T T
3 060-079 150 150 150 276 276 276 F F F
4 080-099 141 141 141 262 262 262 T T T
5 100-119 148 148 148 272 272 272 F F F
6 120-139 150 150 150 275 275 275 F F F
7 140-159 150 150 150 275 275 275 F F F
8 160-179 151 151 151 277 277 277 T T T
9 180-199 148 148 148 272 272 272 T T T
10 200-219 150 150 150 275 275 275 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 148 148 148 272 272 272 F F F
13 260-279 152 152 152 278 278 278 F F F
14 280-299 146 146 146 270 270 270 F F F
15 300-319 142 142 142 265 265 265 F F F
16 320-339 152 152 152 278 278 278 F F F
17 340-359 150 150 150 276 276 276 T T T
18 360-379 151 151 151 277 277 277 F F F
19 380-399 151 151 151 277 277 277 T T T
20 400-419 151 151 151 277 277 277 T T T
21 420-439 150 150 150 275 275 275 F F F
22 440-459 149 149 149 273 273 273 F F F
23 460-479 147 147 147 271 271 271 F F F
24 480-499 150 150 150 274 274 274 F F F
25 500-519 149 149 149 273 273 273 F F F
26 520-539 150 150 150 275 275 275 T T T
27 540-559 149 149 149 273 273 273 T T T
28 560-579 146 146 146 270 270 270 F F F
29 580-599 150 150 150 276 276 276 T T T
Max gen: 152, Max flush: 278
A Difference in extent metadata is expected here
On loop 76
Verifying data now
No change after verify
Loop: 76 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 151 151 151 277 277 277 F F F
1 020-039 144 144 144 267 267 267 F F F
2 040-059 153 153 153 279 279 279 F F F
3 060-079 150 150 150 276 276 276 F F F
4 080-099 153 153 153 279 279 279 F F F
5 100-119 148 148 148 272 272 272 F F F
6 120-139 150 150 150 275 275 275 F F F
7 140-159 150 150 150 275 275 275 F F F
8 160-179 153 153 153 279 279 279 F F F
9 180-199 153 153 153 279 279 279 F F F
10 200-219 150 150 150 275 275 275 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 148 148 148 272 272 272 F F F
13 260-279 152 152 152 278 278 278 F F F
14 280-299 146 146 146 270 270 270 F F F
15 300-319 142 142 142 265 265 265 F F F
16 320-339 152 152 152 278 278 278 F F F
17 340-359 153 153 153 279 279 279 F F F
18 360-379 151 151 151 277 277 277 F F F
19 380-399 153 153 153 279 279 279 F F F
20 400-419 153 153 153 279 279 279 F F F
21 420-439 150 150 150 275 275 275 F F F
22 440-459 149 149 149 273 273 273 F F F
23 460-479 147 147 147 271 271 271 F F F
24 480-499 150 150 150 274 274 274 F F F
25 500-519 149 149 149 273 273 273 F F F
26 520-539 153 153 153 279 279 279 F F F
27 540-559 153 153 153 279 279 279 F F F
28 560-579 146 146 146 270 270 270 F F F
29 580-599 153 153 153 279 279 279 F F F
Max gen: 153, Max flush: 279
Begin loop 77
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.502707884Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.548720181Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.549726258Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.549780514Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.549798783Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.549817032Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798}
{"msg":"Crucible f7e827c1-9be7-45ab-9252-d748eefa2f57 has session id: 30538fb1-32bc-4651-a895-0aeb5dc460c7","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.550414172Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"Upstairs opts: Upstairs UUID: f7e827c1-9be7-45ab-9252-d748eefa2f57, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.550457375Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"Crucible stats registered with UUID: f7e827c1-9be7-45ab-9252-d748eefa2f57","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.550480811Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:09.550501248Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.550973486Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"task":"crutest"}
{"msg":"The guest has requested activation with gen:154","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.551008124Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798}
{"msg":"f7e827c1-9be7-45ab-9252-d748eefa2f57 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.551075772Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.551093141Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"0","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.551121594Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"0","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.551135275Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"1","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.551149007Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"1","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.551161899Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"2","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.55117557Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"2","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.551194109Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.551217484Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"io task","client":"0","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.551232875Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"io task","client":"1","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"ds_connection connected from Ok(127.0.0.1:64891)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.551307529Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"io task","client":"2","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50405)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.551342697Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"io task","client":"1","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49734)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.551373388Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"io task","client":"0","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.552045232Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"2","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.552073914Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"0","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.552264976Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"1","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.552748706Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"0","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.552781376Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"1","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.552809039Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"2","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553364595Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553390249Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"[0]R flush_numbers[0..12]: [277, 267, 279, 276, 279, 272, 275, 275, 279, 279, 275, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.55342141Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"downstairs","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"[0]R generation[0..12]: [151, 144, 153, 150, 153, 148, 150, 150, 153, 153, 150, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.55343668Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"downstairs","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553450722Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"downstairs","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"[1]R flush_numbers[0..12]: [277, 267, 279, 276, 279, 272, 275, 275, 279, 279, 275, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553464143Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"downstairs","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"[1]R generation[0..12]: [151, 144, 153, 150, 153, 148, 150, 150, 153, 153, 150, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553477505Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"downstairs","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553491127Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"downstairs","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"[2]R flush_numbers[0..12]: [277, 267, 279, 276, 279, 272, 275, 275, 279, 279, 275, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553504598Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"downstairs","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"[2]R generation[0..12]: [151, 144, 153, 150, 153, 148, 150, 150, 153, 153, 150, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.55351778Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"downstairs","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553531262Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"downstairs","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"Max found gen is 154","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553547122Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"downstairs","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"Generation requested: 154 >= found:154","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553560164Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"downstairs","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"Next flush: 280","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553572846Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"downstairs","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553585378Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"":"downstairs","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.55359839Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:43:09.553610063Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"0","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553629361Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"1","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553642063Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"client":"2","":"downstairs","session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553654665Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553666568Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"f7e827c1-9be7-45ab-9252-d748eefa2f57 is now active with session: 30538fb1-32bc-4651-a895-0aeb5dc460c7","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.55367915Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553690693Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798,"session_id":"30538fb1-32bc-4651-a895-0aeb5dc460c7"}
{"msg":"The guest has finished waiting for activation with:154","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:09.553702446Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 126 len 28672 data: 8 10 10 11 9 10 8
02/30 Read block 379 len 40960
03/30 Read block 9 len 20480
04/30 Read block 135 len 24576
05/30 Read block 237 len 16384
06/30 Flush
07/30 Write block 116 len 36864 data: 16 19 16 18 16 15 14 10 10
08/30 Read block 260 len 28672
09/30 Read block 522 len 24576
10/30 Read block 138 len 32768
11/30 Read block 308 len 32768
12/30 Flush
13/30 Write block 433 len 20480 data: 13 15 18 15 10
14/30 Read block 334 len 40960
15/30 Read block 224 len 8192
16/30 Write block 246 len 32768 data: 16 17 16 13 10 8 8 10
17/30 Write block 325 len 16384 data: 18 17 21 16
18/30 Write block 404 len 12288 data: 13 14 12
19/30 Write block 167 len 8192 data: 16 12
20/30 Write block 418 len 32768 data: 15 15 15 12 13 14 11 10
21/30 Write block 94 len 40960 data: 16 16 14 13 15 13 15 16 17 18
22/30 Read block 450 len 4096
23/30 Read block 153 len 8192
24/30 Write block 278 len 12288 data: 17 15 17
25/30 Read block 500 len 16384
26/30 Write block 208 len 20480 data: 13 11 8 10 10
27/30 Write block 420 len 32768 data: 16 13 14 15 12 11 10 10
28/30 Write block 454 len 16384 data: 9 13 9 8
29/30 Write block 410 len 24576 data: 11 11 10 11 9 8
30/30 Read block 254 len 20480
----------------------------------------------------------------
Crucible gen:154 GIO:true work queues: Upstairs:0 downstairs:18
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1012 Acked Write 5 Done Done Done false
1013 Acked Read 10 Done Done Done false
1014 Acked Read 2 Done Done Done false
1015 Acked Write 8 Done Done Done false
1016 Acked Write 4 Done Done Done false
1017 Acked Write 3 Done Done Done false
1018 Acked Write 2 Done Done Done false
1019 Acked Write 8 Done Done Done false
1020 Acked Write 10 Done Done Done false
1021 Acked Read 1 Done Done Done false
1022 Acked Read 2 Done Done Done false
1023 Acked Write 3 Done Done Done false
1024 Acked Read 4 Done Done Done false
1025 Acked Write 5 Done Done Done false
1026 Acked Write 8 Done Done Done false
1027 Acked Write 4 Done Done Done false
1028 Acked Write 6 Done Done Sent false
1029 Acked Read 5 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 2 3
Done 18 17 16 51
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1011 1011 1011
Downstairs last five completed: 1011 1010 1009 1008 1007
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:09.573838177Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1798}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 151 151 151 277 277 277 F F F
1 020-039 144 144 144 267 267 267 F F F
2 040-059 153 153 153 279 279 279 F F F
3 060-079 150 150 150 276 276 276 F F F
4 080-099 153 153 153 279 279 279 T T T
5 100-119 154 154 154 281 281 281 T T T
6 120-139 154 154 154 281 281 281 F F F
7 140-159 150 150 150 275 275 275 F F F
8 160-179 153 153 153 279 279 279 T T T
9 180-199 153 153 153 279 279 279 F F F
10 200-219 150 150 150 275 275 275 T T T
11 220-239 146 146 146 269 269 269 F F F
12 240-259 148 148 148 272 272 272 T T T
13 260-279 152 152 152 278 278 278 T T T
14 280-299 146 146 146 270 270 270 T T T
15 300-319 142 142 142 265 265 265 F F F
16 320-339 152 152 152 278 278 278 T T T
17 340-359 153 153 153 279 279 279 F F F
18 360-379 151 151 151 277 277 277 F F F
19 380-399 153 153 153 279 279 279 F F F
20 400-419 153 153 153 279 279 279 T T T
21 420-439 150 150 150 275 275 275 T T T
22 440-459 149 149 149 273 273 273 T T T
23 460-479 147 147 147 271 271 271 F F F
24 480-499 150 150 150 274 274 274 F F F
25 500-519 149 149 149 273 273 273 F F F
26 520-539 153 153 153 279 279 279 F F F
27 540-559 153 153 153 279 279 279 F F F
28 560-579 146 146 146 270 270 270 F F F
29 580-599 153 153 153 279 279 279 F F F
Max gen: 154, Max flush: 281
A Difference in extent metadata is expected here
On loop 77
Verifying data now
No change after verify
Loop: 77 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 151 151 151 277 277 277 F F F
1 020-039 144 144 144 267 267 267 F F F
2 040-059 153 153 153 279 279 279 F F F
3 060-079 150 150 150 276 276 276 F F F
4 080-099 155 155 155 282 282 282 F F F
5 100-119 155 155 155 282 282 282 F F F
6 120-139 154 154 154 281 281 281 F F F
7 140-159 150 150 150 275 275 275 F F F
8 160-179 155 155 155 282 282 282 F F F
9 180-199 153 153 153 279 279 279 F F F
10 200-219 155 155 155 282 282 282 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 155 155 155 282 282 282 F F F
13 260-279 155 155 155 282 282 282 F F F
14 280-299 155 155 155 282 282 282 F F F
15 300-319 142 142 142 265 265 265 F F F
16 320-339 155 155 155 282 282 282 F F F
17 340-359 153 153 153 279 279 279 F F F
18 360-379 151 151 151 277 277 277 F F F
19 380-399 153 153 153 279 279 279 F F F
20 400-419 155 155 155 282 282 282 F F F
21 420-439 155 155 155 282 282 282 F F F
22 440-459 155 155 155 282 282 282 F F F
23 460-479 147 147 147 271 271 271 F F F
24 480-499 150 150 150 274 274 274 F F F
25 500-519 149 149 149 273 273 273 F F F
26 520-539 153 153 153 279 279 279 F F F
27 540-559 153 153 153 279 279 279 F F F
28 560-579 146 146 146 270 270 270 F F F
29 580-599 153 153 153 279 279 279 F F F
Max gen: 155, Max flush: 282
Begin loop 78
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.866039289Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.913554835Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.914721553Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.914777368Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.914799665Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.914821951Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806}
{"msg":"Crucible ab263fb7-67cd-4dd7-9018-59086535411f has session id: 496930a3-1caf-4037-957d-c4e63ae62007","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.915510294Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"Upstairs opts: Upstairs UUID: ab263fb7-67cd-4dd7-9018-59086535411f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.915557305Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"Crucible stats registered with UUID: ab263fb7-67cd-4dd7-9018-59086535411f","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.915575934Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:13.915589476Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.91595416Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"task":"crutest"}
{"msg":"The guest has requested activation with gen:156","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.915985131Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806}
{"msg":"ab263fb7-67cd-4dd7-9018-59086535411f active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.91605028Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.916069008Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"0","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.916093333Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"0","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.916107345Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"1","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.916121976Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"1","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.916143213Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"2","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.916172704Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"2","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.916197559Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"io task","client":"2","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.916221794Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.916253734Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"ds_connection connected from Ok(127.0.0.1:56360)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.916278399Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"io task","client":"0","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"ds_connection connected from Ok(127.0.0.1:40623)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.916418602Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"io task","client":"1","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"ds_connection connected from Ok(127.0.0.1:33367)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.916468701Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"io task","client":"2","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.917190014Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"0","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.917217927Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"1","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.917232408Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"2","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.91792406Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"1","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.917952912Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"2","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.917982913Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"0","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918599401Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918625396Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"[0]R flush_numbers[0..12]: [277, 267, 279, 276, 282, 282, 281, 275, 282, 279, 282, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918707155Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"downstairs","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"[0]R generation[0..12]: [151, 144, 153, 150, 155, 155, 154, 150, 155, 153, 155, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918726623Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"downstairs","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918744582Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"downstairs","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"[1]R flush_numbers[0..12]: [277, 267, 279, 276, 282, 282, 281, 275, 282, 279, 282, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918758533Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"downstairs","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"[1]R generation[0..12]: [151, 144, 153, 150, 155, 155, 154, 150, 155, 153, 155, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918771935Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"downstairs","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918785796Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"downstairs","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"[2]R flush_numbers[0..12]: [277, 267, 279, 276, 282, 282, 281, 275, 282, 279, 282, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918799048Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"downstairs","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"[2]R generation[0..12]: [151, 144, 153, 150, 155, 155, 154, 150, 155, 153, 155, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.91881222Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"downstairs","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918825761Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"downstairs","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"Max found gen is 156","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918838084Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"downstairs","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"Generation requested: 156 >= found:156","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918850886Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"downstairs","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"Next flush: 283","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918863268Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"Wait for a query_work_queue command to finish before sending IO
downstairs","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918882696Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"":"downstairs","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918895538Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918907231Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"0","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918920023Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"1","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918932465Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"client":"2","":"downstairs","session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918947966Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918960188Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"ab263fb7-67cd-4dd7-9018-59086535411f is now active with session: 496930a3-1caf-4037-957d-c4e63ae62007","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.91897271Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918984483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806,"session_id":"496930a3-1caf-4037-957d-c4e63ae62007"}
{"msg":"The guest has finished waiting for activation with:156","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:13.918996406Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 71 len 28672 data: 19 15 16 15 13 10 8
02/30 Read block 559 len 24576
03/30 Read block 509 len 40960
04/30 Write block 1 len 36864 data: 4 5 9 10 11 12 14 11 12
05/30 Write block 322 len 32768 data: 11 14 15 19 18 22 17 14
06/30 Read block 177 len 28672
07/30 Read block 398 len 32768
08/30 Flush
09/30 Flush
10/30 Flush
11/30 Write block 201 len 24576 data: 8 10 10 12 12 12
12/30 Write block 208 len 24576 data: 14 12 9 11 11 10
13/30 Read block 185 len 40960
14/30 Read block 278 len 36864
15/30 Read block 1 len 12288
16/30 Read block 140 len 16384
17/30 Read block 294 len 24576
18/30 Read block 247 len 8192
19/30 Read block 478 len 28672
20/30 Write block 587 len 24576 data: 10 11 13 13 11 11
21/30 Read block 242 len 4096
22/30 Read block 325 len 40960
23/30 Read block 175 len 40960
24/30 Read block 44 len 28672
25/30 Write block 567 len 36864 data: 9 9 9 11 11 12 13 12 14
26/30 Read block 187 len 40960
27/30 Read block 540 len 40960
28/30 Write block 500 len 28672 data: 8 6 7 6 7 8 9
29/30 Write block 18 len 28672 data: 10 10 11 12 13 15 17
30/30 Write block 74 len 24576 data: 16 14 11 9 10 9
----------------------------------------------------------------
Crucible gen:156 GIO:true work queues: Upstairs:0 downstairs:20
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1010 Acked Write 6 Done Done Done false
1011 Acked Write 6 Done Done Done false
1012 Acked Read 10 Done Done Done false
1013 Acked Read 9 Done Done Done false
1014 Acked Read 3 Done Done Done false
1015 Acked Read 4 Done Done Done false
1016 Acked Read 6 Done Done Done false
1017 Acked Read 2 Done Done Done false
1018 Acked Read 7 Done Done Done false
1019 Acked Write 6 Done Done Done false
1020 Acked Read 1 Done Done Done false
1021 Acked Read 10 Done Done Done false
1022 Acked Read 10 Done Done Done false
1023 Acked Read 7 Done Done Done false
1024 Acked Write 9 Done Done Done false
1025 Acked Read 10 Done Done Done false
1026 Acked Read 10 Done Done Done false
1027 Acked Write 7 Done Done Done false
1028 Acked Write 7 Sent Done Sent false
1029 Acked Write 6 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 1 2 5
Done 18 19 18 55
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1009 1009 1009
Downstairs last five completed: 1009 1008 1007 1006 1005
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:13.942150198Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1806}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 156 156 156 283 283 283 T T T
1 020-039 144 144 144 267 267 267 T T T
2 040-059 153 153 153 279 279 279 F F F
3 060-079 156 156 156 283 283 283 T T T
4 080-099 155 155 155 282 282 282 F F F
5 100-119 155 155 155 282 282 282 F F F
6 120-139 154 154 154 281 281 281 F F F
7 140-159 150 150 150 275 275 275 F F F
8 160-179 155 155 155 282 282 282 F F F
9 180-199 153 153 153 279 279 279 F F F
10 200-219 155 155 155 282 282 282 T T T
11 220-239 146 146 146 269 269 269 F F F
12 240-259 155 155 155 282 282 282 F F F
13 260-279 155 155 155 282 282 282 F F F
14 280-299 155 155 155 282 282 282 F F F
15 300-319 142 142 142 265 265 265 F F F
16 320-339 156 156 156 283 283 283 F F F
17 340-359 153 153 153 279 279 279 F F F
18 360-379 151 151 151 277 277 277 F F F
19 380-399 153 153 153 279 279 279 F F F
20 400-419 155 155 155 282 282 282 F F F
21 420-439 155 155 155 282 282 282 F F F
22 440-459 155 155 155 282 282 282 F F F
23 460-479 147 147 147 271 271 271 F F F
24 480-499 150 150 150 274 274 274 F F F
25 500-519 149 149 149 273 273 273 T T T
26 520-539 153 153 153 279 279 279 F F F
27 540-559 153 153 153 279 279 279 F F F
28 560-579 146 146 146 270 270 270 T T T
29 580-599 153 153 153 279 279 279 T T T
Max gen: 156, Max flush: 283
A Difference in extent metadata is expected here
On loop 78
Verifying data now
No change after verify
Loop: 78 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 157 157 157 284 284 284 F F F
1 020-039 157 157 157 284 284 284 F F F
2 040-059 153 153 153 279 279 279 F F F
3 060-079 157 157 157 284 284 284 F F F
4 080-099 155 155 155 282 282 282 F F F
5 100-119 155 155 155 282 282 282 F F F
6 120-139 154 154 154 281 281 281 F F F
7 140-159 150 150 150 275 275 275 F F F
8 160-179 155 155 155 282 282 282 F F F
9 180-199 153 153 153 279 279 279 F F F
10 200-219 157 157 157 284 284 284 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 155 155 155 282 282 282 F F F
13 260-279 155 155 155 282 282 282 F F F
14 280-299 155 155 155 282 282 282 F F F
15 300-319 142 142 142 265 265 265 F F F
16 320-339 156 156 156 283 283 283 F F F
17 340-359 153 153 153 279 279 279 F F F
18 360-379 151 151 151 277 277 277 F F F
19 380-399 153 153 153 279 279 279 F F F
20 400-419 155 155 155 282 282 282 F F F
21 420-439 155 155 155 282 282 282 F F F
22 440-459 155 155 155 282 282 282 F F F
23 460-479 147 147 147 271 271 271 F F F
24 480-499 150 150 150 274 274 274 F F F
25 500-519 157 157 157 284 284 284 F F F
26 520-539 153 153 153 279 279 279 F F F
27 540-559 153 153 153 279 279 279 F F F
28 560-579 157 157 157 284 284 284 F F F
29 580-599 157 157 157 284 284 284 F F F
Max gen: 157, Max flush: 284
Begin loop 79
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.221668634Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.268284517Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.269419914Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.269469334Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.269484834Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.269499685Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814}
{"msg":"Crucible d8335ea0-13d2-43f0-a322-dd3f7c2917f9 has session id: 01e86761-036e-4627-bf66-c671926a35cb","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.270092907Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"Upstairs opts: Upstairs UUID: d8335ea0-13d2-43f0-a322-dd3f7c2917f9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.270130924Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"Crucible stats registered with UUID: d8335ea0-13d2-43f0-a322-dd3f7c2917f9","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.270148343Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:18.270161065Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.270628796Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"task":"crutest"}
{"msg":"The guest has requested activation with gen:158","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.270659077Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814}
{"msg":"d8335ea0-13d2-43f0-a322-dd3f7c2917f9 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.270730033Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.270747872Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"0","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.270778633Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"0","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.270793363Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"1","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.270807315Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"1","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.270872345Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"2","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.270915538Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"2","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.270952405Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"io task","client":"2","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.27097646Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.271005172Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"io task","client":"1","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"ds_connection connected from Ok(127.0.0.1:60284)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.271028918Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"io task","client":"0","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53844)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.271050674Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"io task","client":"2","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"ds_connection connected from Ok(127.0.0.1:59351)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.271207447Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"io task","client":"1","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.271795873Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"1","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.271825235Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"0","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.272486545Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"1","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.272518525Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"2","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.2725593Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"0","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.27312506Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273169722Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273183423Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"2","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"[0]R flush_numbers[0..12]: [284, 284, 279, 284, 282, 282, 281, 275, 282, 279, 284, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273712386Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"downstairs","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"[0]R generation[0..12]: [157, 157, 153, 157, 155, 155, 154, 150, 155, 153, 157, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273736461Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"downstairs","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273751242Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"downstairs","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"[1]R flush_numbers[0..12]: [284, 284, 279, 284, 282, 282, 281, 275, 282, 279, 284, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273764963Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"downstairs","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"[1]R generation[0..12]: [157, 157, 153, 157, 155, 155, 154, 150, 155, 153, 157, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273778325Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"downstairs","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273791877Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"downstairs","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"[2]R flush_numbers[0..12]: [284, 284, 279, 284, 282, 282, 281, 275, 282, 279, 284, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273805109Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"downstairs","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"[2]R generation[0..12]: [157, 157, 153, 157, 155, 155, 154, 150, 155, 153, 157, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273818201Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"downstairs","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273831522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"downstairs","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"Max found gen is 158","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273847892Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"downstairs","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"Wait for a query_work_queue command to finish before sending IO
Generation requested: 158 >= found:158","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.27386766Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"downstairs","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"Next flush: 285","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273880562Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"downstairs","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273893044Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"":"downstairs","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273905666Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273917239Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"0","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273930011Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"1","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273942553Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"client":"2","":"downstairs","session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273955186Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273966958Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"d8335ea0-13d2-43f0-a322-dd3f7c2917f9 is now active with session: 01e86761-036e-4627-bf66-c671926a35cb","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.273979501Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.274007383Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814,"session_id":"01e86761-036e-4627-bf66-c671926a35cb"}
{"msg":"The guest has finished waiting for activation with:158","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:18.274021484Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 581 len 16384 data: 8 6 5 4
02/30 Read block 43 len 20480
03/30 Read block 507 len 24576
04/30 Read block 446 len 16384
05/30 Write block 91 len 12288 data: 16 19 21
06/30 Write block 124 len 20480 data: 11 7 9 11 11
07/30 Read block 93 len 4096
08/30 Write block 309 len 4096 data: 15
09/30 Read block 341 len 12288
10/30 Read block 184 len 20480
11/30 Write block 185 len 20480 data: 13 11 13 8 7
12/30 Flush
13/30 Write block 207 len 32768 data: 13 15 13 10 12 12 11 10
14/30 Read block 191 len 12288
15/30 Read block 525 len 28672
16/30 Read block 396 len 40960
17/30 Read block 42 len 16384
18/30 Read block 393 len 4096
19/30 Flush
20/30 Flush
21/30 Flush
22/30 Write block 489 len 20480 data: 11 13 15 19 18
23/30 Flush
24/30 Write block 421 len 20480 data: 14 15 16 13 12
25/30 Flush
26/30 Write block 354 len 24576 data: 11 9 11 12 14 15
27/30 Write block 166 len 28672 data: 18 17 13 9 12 11 10
28/30 Write block 515 len 40960 data: 10 10 9 10 11 11 13 15 13 13
29/30 Read block 215 len 8192
30/30 Write block 15 len 4096 data: 4
----------------------------------------------------------------
Crucible gen:158 GIO:true work queues: Upstairs:0 downstairs:5
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1025 Acked Write 6 Done Done Done false
1026 Acked Write 7 Done Done Done false
1027 Acked Write 10 Done Done Sent false
1028 Acked Read 2 Done Done Sent false
1029 Acked Write 1 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 3 5
Done 4 4 2 10
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1024 1024 1024
Downstairs last five completed: 1024 1023 1022 1021 1020
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:18.293263779Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1814}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 157 157 157 284 284 284 T T T
1 020-039 157 157 157 284 284 284 F F F
2 040-059 153 153 153 279 279 279 F F F
3 060-079 157 157 157 284 284 284 F F F
4 080-099 158 158 158 285 285 285 F F F
5 100-119 155 155 155 282 282 282 F F F
6 120-139 158 158 158 285 285 285 F F F
7 140-159 150 150 150 275 275 275 F F F
8 160-179 155 155 155 282 282 282 T T T
9 180-199 158 158 158 285 285 285 F F F
10 200-219 158 158 158 286 286 286 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 155 155 155 282 282 282 F F F
13 260-279 155 155 155 282 282 282 F F F
14 280-299 155 155 155 282 282 282 F F F
15 300-319 158 158 158 285 285 285 F F F
16 320-339 156 156 156 283 283 283 F F F
17 340-359 153 153 153 279 279 279 T T T
18 360-379 151 151 151 277 277 277 F F F
19 380-399 153 153 153 279 279 279 F F F
20 400-419 155 155 155 282 282 282 F F F
21 420-439 158 158 158 290 290 290 F F F
22 440-459 155 155 155 282 282 282 F F F
23 460-479 147 147 147 271 271 271 F F F
24 480-499 158 158 158 289 289 289 F F F
25 500-519 157 157 157 284 284 284 T T T
26 520-539 153 153 153 279 279 279 T T T
27 540-559 153 153 153 279 279 279 F F F
28 560-579 157 157 157 284 284 284 F F F
29 580-599 158 158 158 285 285 285 F F F
Max gen: 158, Max flush: 290
A Difference in extent metadata is expected here
On loop 79
Verifying data now
No change after verify
Loop: 79 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 159 159 159 291 291 291 F F F
1 020-039 157 157 157 284 284 284 F F F
2 040-059 153 153 153 279 279 279 F F F
3 060-079 157 157 157 284 284 284 F F F
4 080-099 158 158 158 285 285 285 F F F
5 100-119 155 155 155 282 282 282 F F F
6 120-139 158 158 158 285 285 285 F F F
7 140-159 150 150 150 275 275 275 F F F
8 160-179 159 159 159 291 291 291 F F F
9 180-199 158 158 158 285 285 285 F F F
10 200-219 158 158 158 286 286 286 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 155 155 155 282 282 282 F F F
13 260-279 155 155 155 282 282 282 F F F
14 280-299 155 155 155 282 282 282 F F F
15 300-319 158 158 158 285 285 285 F F F
16 320-339 156 156 156 283 283 283 F F F
17 340-359 159 159 159 291 291 291 F F F
18 360-379 151 151 151 277 277 277 F F F
19 380-399 153 153 153 279 279 279 F F F
20 400-419 155 155 155 282 282 282 F F F
21 420-439 158 158 158 290 290 290 F F F
22 440-459 155 155 155 282 282 282 F F F
23 460-479 147 147 147 271 271 271 F F F
24 480-499 158 158 158 289 289 289 F F F
25 500-519 159 159 159 291 291 291 F F F
26 520-539 159 159 159 291 291 291 F F F
27 540-559 153 153 153 279 279 279 F F F
28 560-579 157 157 157 284 284 284 F F F
29 580-599 158 158 158 285 285 285 F F F
Max gen: 159, Max flush: 291
Begin loop 80
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.552382738Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.597727248Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.598764336Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.598815365Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.598834373Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.598852542Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822}
{"msg":"Crucible ac423617-ee2f-4f52-9d02-e5c5ac43c0e4 has session id: 7661e32c-5860-45e3-b0f1-70eef0529b65","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.599397244Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"Upstairs opts: Upstairs UUID: ac423617-ee2f-4f52-9d02-e5c5ac43c0e4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.599495513Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"Crucible stats registered with UUID: ac423617-ee2f-4f52-9d02-e5c5ac43c0e4","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.599527284Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:22.599554587Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.60011414Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"task":"crutest"}
{"msg":"The guest has requested activation with gen:160","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.600144871Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822}
{"msg":"ac423617-ee2f-4f52-9d02-e5c5ac43c0e4 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.600232737Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.600250056Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"0","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.600274051Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"0","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.600287693Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"1","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.600301944Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"1","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.600314886Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"2","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.600328568Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"2","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.600341229Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.600354501Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"io task","client":"0","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.600367593Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"io task","client":"1","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57046)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.600606925Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"io task","client":"1","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39810)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.600645511Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"io task","client":"2","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"ds_connection connected from Ok(127.0.0.1:39381)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.60068012Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"io task","client":"0","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.601489788Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"0","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.601530024Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"1","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.602246809Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"0","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.602366455Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"1","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.602388192Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"2","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.602855803Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.602896167Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603144864Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"2","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"[0]R flush_numbers[0..12]: [291, 284, 279, 284, 285, 282, 285, 275, 291, 285, 286, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603670818Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"downstairs","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"[0]R generation[0..12]: [159, 157, 153, 157, 158, 155, 158, 150, 159, 158, 158, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603696872Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"downstairs","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603711662Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"downstairs","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"[1]R flush_numbers[0..12]: [291, 284, 279, 284, 285, 282, 285, 275, 291, 285, 286, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603725114Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"downstairs","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"[1]R generation[0..12]: [159, 157, 153, 157, 158, 155, 158, 150, 159, 158, 158, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603738816Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"downstairs","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603755955Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"downstairs","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"[2]R flush_numbers[0..12]: [291, 284, 279, 284, 285, 282, 285, 275, 291, 285, 286, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603769587Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"downstairs","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"[2]R generation[0..12]: [159, 157, 153, 157, 158, 155, 158, 150, 159, 158, 158, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603782848Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"downstairs","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.60379618Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"downstairs","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"Max found gen is 160","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603808732Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"downstairs","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"Generation requested: 160 >= found:160","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603821374Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"downstairs","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"Next flush: 292","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603833947Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822Wait for a query_work_queue command to finish before sending IO
,"":"downstairs","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603856253Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"":"downstairs","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603868975Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603880628Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"0","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.60389339Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"1","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603905922Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"client":"2","":"downstairs","session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603918884Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603930767Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"ac423617-ee2f-4f52-9d02-e5c5ac43c0e4 is now active with session: 7661e32c-5860-45e3-b0f1-70eef0529b65","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603943159Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603956741Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822,"session_id":"7661e32c-5860-45e3-b0f1-70eef0529b65"}
{"msg":"The guest has finished waiting for activation with:160","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:22.603968903Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 156 len 32768 data: 12 13 14 13 10 15 14 12
02/30 Write block 466 len 36864 data: 9 9 8 7 8 10 9 11 11
03/30 Read block 182 len 4096
04/30 Write block 26 len 16384 data: 13 13 13 13
05/30 Write block 113 len 4096 data: 12
06/30 Write block 447 len 12288 data: 9 9 11
07/30 Read block 336 len 12288
08/30 Read block 75 len 24576
09/30 Read block 282 len 16384
10/30 Flush
11/30 Write block 25 len 12288 data: 14 14 14
12/30 Read block 405 len 4096
13/30 Write block 162 len 20480 data: 15 13 17 17 19
14/30 Read block 196 len 4096
15/30 Write block 259 len 8192 data: 10 12
16/30 Flush
17/30 Write block 175 len 20480 data: 15 14 13 14 15
18/30 Read block 379 len 28672
19/30 Write block 350 len 32768 data: 16 15 13 12 12 10 12 13
20/30 Write block 200 len 24576 data: 8 9 11 11 13 13
21/30 Read block 336 len 16384
22/30 Read block 300 len 40960
23/30 Write block 461 len 4096 data: 10
24/30 Read block 46 len 4096
25/30 Read block 409 len 4096
26/30 Read block 191 len 8192
27/30 Read block 22 len 4096
28/30 Read block 45 len 8192
29/30 Write block 552 len 36864 data: 13 10 12 12 15 13 13 14 13
30/30 Write block 366 len 36864 data: 9 9 13 14 15 15 14 17 18
----------------------------------------------------------------
Crucible gen:160 GIO:true work queues: Upstairs:0 downstairs:14
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1016 Acked Write 5 Done Done Done false
1017 Acked Read 7 Done Done Done false
1018 Acked Write 8 Done Done Done false
1019 Acked Write 6 Done Done Done false
1020 Acked Read 4 Done Done Done false
1021 Acked Read 10 Done Done Done false
1022 Acked Write 1 Done Done Done false
1023 Acked Read 1 Done Done Done false
1024 Acked Read 1 Done Done Done false
1025 Acked Read 2 Done Done Done false
1026 Acked Read 1 Done Done Done false
1027 Acked Read 2 Done Done Done false
1028 Acked Write 9 Sent Sent Sent false
1029 Acked Write 9 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 12 12 12 36
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1015 1015 1015
Downstairs last five completed: 1015 1014 1013 1012 1011
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:22.622193118Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1822}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 159 159 159 291 291 291 F F F
1 020-039 160 160 160 293 293 293 F F F
2 040-059 153 153 153 279 279 279 F F F
3 060-079 157 157 157 284 284 284 F F F
4 080-099 158 158 158 285 285 285 F F F
5 100-119 160 160 160 292 292 292 F F F
6 120-139 158 158 158 285 285 285 F F F
7 140-159 160 160 160 292 292 292 F F F
8 160-179 160 160 160 293 293 293 T T T
9 180-199 158 158 158 285 285 285 F F F
10 200-219 158 158 158 286 286 286 T T T
11 220-239 146 146 146 269 269 269 F F F
12 240-259 160 160 160 293 293 293 F F F
13 260-279 160 160 160 293 293 293 F F F
14 280-299 155 155 155 282 282 282 F F F
15 300-319 158 158 158 285 285 285 F F F
16 320-339 156 156 156 283 283 283 F F F
17 340-359 159 159 159 291 291 291 T T T
18 360-379 151 151 151 277 277 277 T T T
19 380-399 153 153 153 279 279 279 F F F
20 400-419 155 155 155 282 282 282 F F F
21 420-439 158 158 158 290 290 290 F F F
22 440-459 160 160 160 292 292 292 F F F
23 460-479 160 160 160 292 292 292 T T T
24 480-499 158 158 158 289 289 289 F F F
25 500-519 159 159 159 291 291 291 F F F
26 520-539 159 159 159 291 291 291 F F F
27 540-559 153 153 153 279 279 279 T T T
28 560-579 157 157 157 284 284 284 T T T
29 580-599 158 158 158 285 285 285 F F F
Max gen: 160, Max flush: 293
A Difference in extent metadata is expected here
On loop 80
Verifying data now
No change after verify
Loop: 80 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 159 159 159 291 291 291 F F F
1 020-039 160 160 160 293 293 293 F F F
2 040-059 153 153 153 279 279 279 F F F
3 060-079 157 157 157 284 284 284 F F F
4 080-099 158 158 158 285 285 285 F F F
5 100-119 160 160 160 292 292 292 F F F
6 120-139 158 158 158 285 285 285 F F F
7 140-159 160 160 160 292 292 292 F F F
8 160-179 161 161 161 294 294 294 F F F
9 180-199 158 158 158 285 285 285 F F F
10 200-219 161 161 161 294 294 294 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 160 160 160 293 293 293 F F F
13 260-279 160 160 160 293 293 293 F F F
14 280-299 155 155 155 282 282 282 F F F
15 300-319 158 158 158 285 285 285 F F F
16 320-339 156 156 156 283 283 283 F F F
17 340-359 161 161 161 294 294 294 F F F
18 360-379 161 161 161 294 294 294 F F F
19 380-399 153 153 153 279 279 279 F F F
20 400-419 155 155 155 282 282 282 F F F
21 420-439 158 158 158 290 290 290 F F F
22 440-459 160 160 160 292 292 292 F F F
23 460-479 161 161 161 294 294 294 F F F
24 480-499 158 158 158 289 289 289 F F F
25 500-519 159 159 159 291 291 291 F F F
26 520-539 159 159 159 291 291 291 F F F
27 540-559 161 161 161 294 294 294 F F F
28 560-579 161 161 161 294 294 294 F F F
29 580-599 158 158 158 285 285 285 F F F
Max gen: 161, Max flush: 294
Begin loop 81
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.898964904Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.946477722Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.947603735Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.947648857Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.947669814Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.947685834Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830}
{"msg":"Crucible 50ec81ff-8015-4861-b1bd-4bebfa4fdea6 has session id: 1a912a2c-40fa-43d4-9c87-e7b6ad28865c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.948394195Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"Upstairs opts: Upstairs UUID: 50ec81ff-8015-4861-b1bd-4bebfa4fdea6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.948433661Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"Crucible stats registered with UUID: 50ec81ff-8015-4861-b1bd-4bebfa4fdea6","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.94845074Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:26.94846783Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.949055476Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"task":"crutest"}
{"msg":"The guest has requested activation with gen:162","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.949090044Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830}
{"msg":"50ec81ff-8015-4861-b1bd-4bebfa4fdea6 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.94914554Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.949173473Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"0","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.949200806Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"0","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.949214887Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"1","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.949229318Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"1","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.94924287Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"2","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.949262088Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"2","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.949278678Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"io task","client":"2","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.949303522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"io task","client":"0","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.949331185Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:63007)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.949359338Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"io task","client":"2","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47914)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.949734535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"io task","client":"1","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:43937)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.94985623Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"io task","client":"0","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.950203135Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"1","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.950242571Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"0","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.950352902Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"2","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.950880476Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"0","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.950992157Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"2","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951025456Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"1","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951523588Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951561384Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"[0]R flush_numbers[0..12]: [291, 293, 279, 284, 285, 292, 285, 292, 294, 285, 294, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951661272Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"downstairs","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"[0]R generation[0..12]: [159, 160, 153, 157, 158, 160, 158, 160, 161, 158, 161, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.95168113Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"downstairs","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951695861Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"downstairs","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"[1]R flush_numbers[0..12]: [291, 293, 279, 284, 285, 292, 285, 292, 294, 285, 294, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951709592Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"downstairs","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"[1]R generation[0..12]: [159, 160, 153, 157, 158, 160, 158, 160, 161, 158, 161, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951723964Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"downstairs","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951737685Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"downstairs","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"[2]R flush_numbers[0..12]: [291, 293, 279, 284, 285, 292, 285, 292, 294, 285, 294, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951751316Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"downstairs","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"[2]R generation[0..12]: [159, 160, 153, 157, 158, 160, 158, 160, 161, 158, 161, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951764458Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"downstairs","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.95177781Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO
1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"Max found gen is 162","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951796698Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"downstairs","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"Generation requested: 162 >= found:162","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.9518096Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"downstairs","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"Next flush: 295","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951821953Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"downstairs","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951834395Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"":"downstairs","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951850965Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951862598Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"0","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.95187539Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"1","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951887942Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"client":"2","":"downstairs","session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951900574Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951912357Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"50ec81ff-8015-4861-b1bd-4bebfa4fdea6 is now active with session: 1a912a2c-40fa-43d4-9c87-e7b6ad28865c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951924699Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951936362Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830,"session_id":"1a912a2c-40fa-43d4-9c87-e7b6ad28865c"}
{"msg":"The guest has finished waiting for activation with:162","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:26.951948105Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 409 len 16384 data: 11 12 12 11
02/30 Read block 184 len 32768
03/30 Read block 28 len 20480
04/30 Read block 109 len 8192
05/30 Read block 419 len 12288
06/30 Write block 360 len 36864 data: 15 12 13 10 9 8 10 10 14
07/30 Write block 124 len 4096 data: 12
08/30 Write block 491 len 4096 data: 16
09/30 Read block 123 len 8192
10/30 Write block 484 len 4096 data: 13
11/30 Write block 101 len 36864 data: 17 18 19 19 19 16 12 12 12
12/30 Flush
13/30 Write block 487 len 24576 data: 15 15 12 14 17 20
14/30 Read block 33 len 40960
15/30 Flush
16/30 Write block 401 len 4096 data: 13
17/30 Flush
18/30 Flush
19/30 Write block 11 len 4096 data: 9
20/30 Read block 392 len 24576
21/30 Read block 336 len 16384
22/30 Write block 93 len 8192 data: 22 17
23/30 Flush
24/30 Write block 36 len 40960 data: 6 6 7 9 11 13 10 9 9 10
25/30 Read block 140 len 4096
26/30 Read block 437 len 4096
27/30 Write block 313 len 28672 data: 10 13 12 10 9 8 10
28/30 Read block 543 len 16384
29/30 Read block 376 len 32768
30/30 Write block 448 len 36864 data: 10 12 9 10 9 9 10 14 10
----------------------------------------------------------------
Crucible gen:162 GIO:true work queues: Upstairs:0 downstairs:7
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1023 Acked Write 10 Done Done Done false
1024 Acked Read 1 Done Done Done false
1025 Acked Read 1 Done Done Done false
1026 Acked Write 7 Done Done Done false
1027 Acked Read 4 Done Done Done false
1028 Acked Read 8 Done Done Done false
1029 Acked Write 9 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 6 6 6 18
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1022 1022 1022
Downstairs last five completed: 1022 1021 1020 1019 1018
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:26.970692118Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1830}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 162 162 162 299 299 299 F F F
1 020-039 160 160 160 293 293 293 T T T
2 040-059 153 153 153 279 279 279 T T T
3 060-079 157 157 157 284 284 284 F F F
4 080-099 162 162 162 299 299 299 F F F
5 100-119 162 162 162 295 295 295 F F F
6 120-139 162 162 162 295 295 295 F F F
7 140-159 160 160 160 292 292 292 F F F
8 160-179 161 161 161 294 294 294 F F F
9 180-199 158 158 158 285 285 285 F F F
10 200-219 161 161 161 294 294 294 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 160 160 160 293 293 293 F F F
13 260-279 160 160 160 293 293 293 F F F
14 280-299 155 155 155 282 282 282 F F F
15 300-319 158 158 158 285 285 285 T T T
16 320-339 156 156 156 283 283 283 F F F
17 340-359 161 161 161 294 294 294 F F F
18 360-379 162 162 162 295 295 295 F F F
19 380-399 153 153 153 279 279 279 F F F
20 400-419 162 162 162 297 297 297 F F F
21 420-439 158 158 158 290 290 290 F F F
22 440-459 160 160 160 292 292 292 T T T
23 460-479 161 161 161 294 294 294 F F F
24 480-499 162 162 162 296 296 296 F F F
25 500-519 159 159 159 291 291 291 F F F
26 520-539 159 159 159 291 291 291 F F F
27 540-559 161 161 161 294 294 294 F F F
28 560-579 161 161 161 294 294 294 F F F
29 580-599 158 158 158 285 285 285 F F F
Max gen: 162, Max flush: 299
A Difference in extent metadata is expected here
On loop 81
Verifying data now
No change after verify
Loop: 81 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 162 162 162 299 299 299 F F F
1 020-039 163 163 163 300 300 300 F F F
2 040-059 163 163 163 300 300 300 F F F
3 060-079 157 157 157 284 284 284 F F F
4 080-099 162 162 162 299 299 299 F F F
5 100-119 162 162 162 295 295 295 F F F
6 120-139 162 162 162 295 295 295 F F F
7 140-159 160 160 160 292 292 292 F F F
8 160-179 161 161 161 294 294 294 F F F
9 180-199 158 158 158 285 285 285 F F F
10 200-219 161 161 161 294 294 294 F F F
11 220-239 146 146 146 269 269 269 F F F
12 240-259 160 160 160 293 293 293 F F F
13 260-279 160 160 160 293 293 293 F F F
14 280-299 155 155 155 282 282 282 F F F
15 300-319 163 163 163 300 300 300 F F F
16 320-339 156 156 156 283 283 283 F F F
17 340-359 161 161 161 294 294 294 F F F
18 360-379 162 162 162 295 295 295 F F F
19 380-399 153 153 153 279 279 279 F F F
20 400-419 162 162 162 297 297 297 F F F
21 420-439 158 158 158 290 290 290 F F F
22 440-459 163 163 163 300 300 300 F F F
23 460-479 161 161 161 294 294 294 F F F
24 480-499 162 162 162 296 296 296 F F F
25 500-519 159 159 159 291 291 291 F F F
26 520-539 159 159 159 291 291 291 F F F
27 540-559 161 161 161 294 294 294 F F F
28 560-579 161 161 161 294 294 294 F F F
29 580-599 158 158 158 285 285 285 F F F
Max gen: 163, Max flush: 300
Begin loop 82
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.222341363Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.268113309Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.269174602Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.269224751Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.269240691Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.269255741Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838}
{"msg":"Crucible a3f5ebb5-c0a2-4f74-afb2-ed2cdefa8391 has session id: ecb8bbbf-248d-4b04-bfa3-111734cbc515","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.269846106Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"Upstairs opts: Upstairs UUID: a3f5ebb5-c0a2-4f74-afb2-ed2cdefa8391, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.269938978Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"Crucible stats registered with UUID: a3f5ebb5-c0a2-4f74-afb2-ed2cdefa8391","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.269966581Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:31.269980243Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270447553Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"task":"crutest"}
{"msg":"The guest has requested activation with gen:164","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270478264Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838}
{"msg":"a3f5ebb5-c0a2-4f74-afb2-ed2cdefa8391 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270546792Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270580721Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"0","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270620216Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"0","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270650888Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"1","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270704384Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"1","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270737744Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"2","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270768944Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"2","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270793769Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270820762Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"io task","client":"0","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270846007Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"io task","client":"1","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"ds_connection connected from Ok(127.0.0.1:33890)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270871481Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"io task","client":"2","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"ds_connection connected from Ok(127.0.0.1:61700)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.270897535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"io task","client":"0","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"ds_connection connected from Ok(127.0.0.1:57043)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.271030952Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"io task","client":"1","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.271633469Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"2","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.271661322Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"0","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.272222704Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"0","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.272370483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"1","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.2724077Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"2","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.272918583Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.272955021Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273095034Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"1","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"[0]R flush_numbers[0..12]: [299, 300, 300, 284, 299, 295, 295, 292, 294, 285, 294, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273638587Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"downstairs","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"[0]R generation[0..12]: [162, 163, 163, 157, 162, 162, 162, 160, 161, 158, 161, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273659385Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"downstairs","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273674505Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"downstairs","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"[1]R flush_numbers[0..12]: [299, 300, 300, 284, 299, 295, 295, 292, 294, 285, 294, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273687927Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"downstairs","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"[1]R generation[0..12]: [162, 163, 163, 157, 162, 162, 162, 160, 161, 158, 161, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273701528Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"downstairs","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.27371532Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"downstairs","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"[2]R flush_numbers[0..12]: [299, 300, 300, 284, 299, 295, 295, 292, 294, 285, 294, 269]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273735307Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"downstairs","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"[2]R generation[0..12]: [162, 163, 163, 157, 162, 162, 162, 160, 161, 158, 161, 146]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273749529Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"downstairs","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273770605Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO
downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"Max found gen is 164","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273792252Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"downstairs","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"Generation requested: 164 >= found:164","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273805564Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"downstairs","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"Next flush: 301","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273818106Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"downstairs","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273830348Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"":"downstairs","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.27384354Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273855273Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"0","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273868045Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"1","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273880608Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"client":"2","":"downstairs","session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273893619Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273905432Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"a3f5ebb5-c0a2-4f74-afb2-ed2cdefa8391 is now active with session: ecb8bbbf-248d-4b04-bfa3-111734cbc515","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273917904Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.273929537Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838,"session_id":"ecb8bbbf-248d-4b04-bfa3-111734cbc515"}
{"msg":"The guest has finished waiting for activation with:164","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:31.27394136Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 2 len 32768 data: 6 10 11 12 13 15 12 13
02/30 Write block 550 len 12288 data: 12 15 14
03/30 Flush
04/30 Write block 267 len 32768 data: 10 11 14 13 15 14 14 18
05/30 Write block 225 len 20480 data: 8 10 9 10 10
06/30 Read block 159 len 36864
07/30 Write block 533 len 32768 data: 16 16 19 15 14 12 13 12
08/30 Write block 503 len 28672 data: 7 8 9 10 9 10 8
09/30 Read block 559 len 12288
10/30 Read block 1 len 20480
11/30 Flush
12/30 Write block 431 len 36864 data: 11 14 14 16 19 16 11 9 9
13/30 Write block 316 len 8192 data: 11 10
14/30 Read block 318 len 32768
15/30 Write block 160 len 36864 data: 11 16 16 14 18 18 20 18 14
16/30 Read block 541 len 32768
17/30 Write block 495 len 20480 data: 18 18 14 14 13
18/30 Write block 358 len 24576 data: 15 16 16 13 14 11
19/30 Read block 433 len 24576
20/30 Flush
21/30 Write block 461 len 32768 data: 11 14 13 13 12 10 10 9
22/30 Read block 223 len 28672
23/30 Read block 391 len 32768
24/30 Read block 211 len 12288
25/30 Flush
26/30 Write block 187 len 36864 data: 14 9 8 8 12 11 13 11 15
27/30 Write block 462 len 32768 data: 15 14 14 13 11 11 10 8
28/30 Write block 222 len 24576 data: 5 8 9 9 11 10
29/30 Write block 273 len 36864 data: 15 19 19 17 17 18 16 18 17
30/30 Read block 62 len 4096
----------------------------------------------------------------
Crucible gen:164 GIO:true work queues: Upstairs:0 downstairs:5
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1025 Acked Write 9 Done Done Done false
1026 Acked Write 8 Done Done Done false
1027 Acked Write 6 Done Done Done false
1028 Acked Write 9 Done Sent Sent false
1029 Acked Read 1 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 2 2 4
Done 5 3 3 11
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1024 1024 1024
Downstairs last five completed: 1024 1023 1022 1021 1020
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:31.295879205Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1838}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 164 164 164 301 301 301 F F F
1 020-039 163 163 163 300 300 300 F F F
2 040-059 163 163 163 300 300 300 F F F
3 060-079 157 157 157 284 284 284 F F F
4 080-099 162 162 162 299 299 299 F F F
5 100-119 162 162 162 295 295 295 F F F
6 120-139 162 162 162 295 295 295 F F F
7 140-159 160 160 160 292 292 292 F F F
8 160-179 164 164 164 303 303 303 F F F
9 180-199 158 158 158 285 285 285 T T T
10 200-219 161 161 161 294 294 294 F F F
11 220-239 164 164 164 302 302 302 T T T
12 240-259 160 160 160 293 293 293 F F F
13 260-279 164 164 164 302 302 302 T T T
14 280-299 155 155 155 282 282 282 T T T
15 300-319 164 164 164 303 303 303 F F F
16 320-339 156 156 156 283 283 283 F F F
17 340-359 164 164 164 303 303 303 F F F
18 360-379 164 164 164 303 303 303 F F F
19 380-399 153 153 153 279 279 279 F F F
20 400-419 162 162 162 297 297 297 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 163 163 163 300 300 300 F F F
23 460-479 164 164 164 304 304 304 T T T
24 480-499 164 164 164 303 303 303 F F F
25 500-519 164 164 164 302 302 302 F F F
26 520-539 164 164 164 302 302 302 F F F
27 540-559 164 164 164 302 302 302 F F F
28 560-579 161 161 161 294 294 294 F F F
29 580-599 158 158 158 285 285 285 F F F
Max gen: 164, Max flush: 304
A Difference in extent metadata is expected here
On loop 82
Verifying data now
No change after verify
Loop: 82 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 164 164 164 301 301 301 F F F
1 020-039 163 163 163 300 300 300 F F F
2 040-059 163 163 163 300 300 300 F F F
3 060-079 157 157 157 284 284 284 F F F
4 080-099 162 162 162 299 299 299 F F F
5 100-119 162 162 162 295 295 295 F F F
6 120-139 162 162 162 295 295 295 F F F
7 140-159 160 160 160 292 292 292 F F F
8 160-179 164 164 164 303 303 303 F F F
9 180-199 165 165 165 305 305 305 F F F
10 200-219 161 161 161 294 294 294 F F F
11 220-239 165 165 165 305 305 305 F F F
12 240-259 160 160 160 293 293 293 F F F
13 260-279 165 165 165 305 305 305 F F F
14 280-299 165 165 165 305 305 305 F F F
15 300-319 164 164 164 303 303 303 F F F
16 320-339 156 156 156 283 283 283 F F F
17 340-359 164 164 164 303 303 303 F F F
18 360-379 164 164 164 303 303 303 F F F
19 380-399 153 153 153 279 279 279 F F F
20 400-419 162 162 162 297 297 297 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 163 163 163 300 300 300 F F F
23 460-479 165 165 165 305 305 305 F F F
24 480-499 164 164 164 303 303 303 F F F
25 500-519 164 164 164 302 302 302 F F F
26 520-539 164 164 164 302 302 302 F F F
27 540-559 164 164 164 302 302 302 F F F
28 560-579 161 161 161 294 294 294 F F F
29 580-599 158 158 158 285 285 285 F F F
Max gen: 165, Max flush: 305
Begin loop 83
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.55138726Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.595492198Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.596406332Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.596450525Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.596466205Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.596480986Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846}
{"msg":"Crucible ab073b1c-cf2f-4522-9e3f-b3d94a7df364 has session id: 9eefb210-2a41-4dd7-9445-82de48ad3fcf","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.596960449Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"Upstairs opts: Upstairs UUID: ab073b1c-cf2f-4522-9e3f-b3d94a7df364, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597006281Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"Crucible stats registered with UUID: ab073b1c-cf2f-4522-9e3f-b3d94a7df364","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597033804Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:35.597054731Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597538361Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"task":"crutest"}
{"msg":"The guest has requested activation with gen:166","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597581315Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846}
{"msg":"ab073b1c-cf2f-4522-9e3f-b3d94a7df364 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597606059Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597626716Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"0","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597665093Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"0","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597695044Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"1","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597725675Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"1","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597755907Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"2","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597787457Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"2","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597816299Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597839935Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"io task","client":"0","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.59786319Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"io task","client":"1","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"ds_connection connected from Ok(127.0.0.1:48296)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597889993Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"io task","client":"2","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50164)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.597918316Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"io task","client":"1","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53084)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.59794383Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"io task","client":"0","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.598531816Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"1","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.59857301Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"0","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.599320388Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"1","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.599475382Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"0","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.599694856Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"2","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.599988584Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.600113616Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.600378313Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"2","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"[0]R flush_numbers[0..12]: [301, 300, 300, 284, 299, 295, 295, 292, 303, 305, 294, 305]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.60093077Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"downstairs","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"[0]R generation[0..12]: [164, 163, 163, 157, 162, 162, 162, 160, 164, 165, 161, 165]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.600953796Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"downstairs","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.600969357Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"downstairs","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"[1]R flush_numbers[0..12]: [301, 300, 300, 284, 299, 295, 295, 292, 303, 305, 294, 305]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.600983158Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"downstairs","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"[1]R generation[0..12]: [164, 163, 163, 157, 162, 162, 162, 160, 164, 165, 161, 165]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.600999708Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"downstairs","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601013619Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"downstairs","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"[2]R flush_numbers[0..12]: [301, 300, 300, 284, 299, 295, 295, 292, 303, 305, 294, 305]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601027221Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"downstairs","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"[2]R generation[0..12]: [164, 163, 163, 157, 162, 162, 162, 160, 164, 165, 161, 165]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601040942Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"downstairs","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601054464Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"downstairs","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"Max found gen is 166","v":0,"name":"crucible","level":Wait for a query_work_queue command to finish before sending IO
30,"time":"2025-01-27T19:43:35.601074441Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"downstairs","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"Generation requested: 166 >= found:166","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601087444Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"downstairs","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"Next flush: 306","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601099976Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"downstairs","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601112368Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"":"downstairs","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.6011253Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601136853Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"0","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601149605Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"1","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601162037Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"client":"2","":"downstairs","session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.60117459Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601186442Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"ab073b1c-cf2f-4522-9e3f-b3d94a7df364 is now active with session: 9eefb210-2a41-4dd7-9445-82de48ad3fcf","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601200623Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601212216Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846,"session_id":"9eefb210-2a41-4dd7-9445-82de48ad3fcf"}
{"msg":"The guest has finished waiting for activation with:166","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:35.601224079Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Flush
02/30 Write block 93 len 28672 data: 23 18 17 15 14 16 14
03/30 Flush
04/30 Write block 245 len 36864 data: 13 17 18 17 14 11 9 9 11
05/30 Read block 199 len 8192
06/30 Read block 256 len 20480
07/30 Read block 306 len 36864
08/30 Read block 448 len 16384
09/30 Read block 384 len 4096
10/30 Read block 391 len 24576
11/30 Write block 359 len 40960 data: 17 17 14 15 12 10 9 11 11 15
12/30 Write block 175 len 24576 data: 16 15 14 15 16 14
13/30 Write block 224 len 4096 data: 10
14/30 Read block 126 len 28672
15/30 Write block 219 len 28672 data: 9 7 5 6 9 11 10
16/30 Read block 401 len 8192
17/30 Write block 143 len 40960 data: 15 14 15 16 15 15 15 13 12 12
18/30 Read block 341 len 32768
19/30 Read block 164 len 36864
20/30 Read block 572 len 32768
21/30 Read block 225 len 8192
22/30 Write block 313 len 36864 data: 11 14 13 12 11 9 11 10 10
23/30 Write block 4 len 16384 data: 12 13 14 16
24/30 Write block 121 len 32768 data: 16 15 11 13 8 10 12 12
25/30 Read block 442 len 4096
26/30 Read block 12 len 28672
27/30 Write block 95 len 24576 data: 18 16 15 17 15 16
28/30 Write block 334 len 40960 data: 5 6 8 9 12 12 12 14 16 16
29/30 Write block 210 len 8192 data: 11 13
30/30 Read block 493 len 36864
----------------------------------------------------------------
Crucible gen:166 GIO:true work queues: Upstairs:0 downstairs:27
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1003 Acked Write 9 Done Done Done false
1004 Acked Read 2 Done Done Done false
1005 Acked Read 5 Done Done Done false
1006 Acked Read 9 Done Done Done false
1007 Acked Read 4 Done Done Done false
1008 Acked Read 1 Done Done Done false
1009 Acked Read 6 Done Done Done false
1010 Acked Write 10 Done Done Done false
1011 Acked Write 6 Done Done Done false
1012 Acked Write 1 Done Done Done false
1013 Acked Read 7 Done Done Done false
1014 Acked Write 7 Done Done Done false
1015 Acked Read 2 Done Done Done false
1016 Acked Write 10 Done Done Done false
1017 Acked Read 8 Done Done Done false
1018 Acked Read 9 Done Done Done false
1019 Acked Read 8 Done Done Done false
1020 Acked Read 2 Done Done Done false
1021 Acked Write 9 Done Done Done false
1022 Acked Write 4 Done Done Done false
1023 Acked Write 8 Done Done Done false
1024 Acked Read 1 Done Done Done false
1025 Acked Read 7 Done Done Done false
1026 Acked Write 6 Done Done Done false
1027 Acked Write 10 Done Done Sent false
1028 Acked Write 2 Done Done Sent false
1029 Acked Read 9 Done Done Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 3 3
Done 27 27 24 78
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1002 1002 1002
Downstairs last five completed: 1002 1001 1000
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:35.623269397Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1846}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 164 164 164 301 301 301 T T T
1 020-039 163 163 163 300 300 300 F F F
2 040-059 163 163 163 300 300 300 F F F
3 060-079 157 157 157 284 284 284 F F F
4 080-099 166 166 166 307 307 307 T T T
5 100-119 162 162 162 295 295 295 T T T
6 120-139 162 162 162 295 295 295 T T T
7 140-159 160 160 160 292 292 292 T T T
8 160-179 164 164 164 303 303 303 T T T
9 180-199 165 165 165 305 305 305 T T T
10 200-219 161 161 161 294 294 294 T T T
11 220-239 165 165 165 305 305 305 T T T
12 240-259 160 160 160 293 293 293 T T T
13 260-279 165 165 165 305 305 305 F F F
14 280-299 165 165 165 305 305 305 F F F
15 300-319 164 164 164 303 303 303 T T T
16 320-339 156 156 156 283 283 283 T T T
17 340-359 164 164 164 303 303 303 T T T
18 360-379 164 164 164 303 303 303 T T T
19 380-399 153 153 153 279 279 279 F F F
20 400-419 162 162 162 297 297 297 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 163 163 163 300 300 300 F F F
23 460-479 165 165 165 305 305 305 F F F
24 480-499 164 164 164 303 303 303 F F F
25 500-519 164 164 164 302 302 302 F F F
26 520-539 164 164 164 302 302 302 F F F
27 540-559 164 164 164 302 302 302 F F F
28 560-579 161 161 161 294 294 294 F F F
29 580-599 158 158 158 285 285 285 F F F
Max gen: 166, Max flush: 307
A Difference in extent metadata is expected here
On loop 83
Verifying data now
No change after verify
Loop: 83 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 167 167 167 308 308 308 F F F
1 020-039 163 163 163 300 300 300 F F F
2 040-059 163 163 163 300 300 300 F F F
3 060-079 157 157 157 284 284 284 F F F
4 080-099 167 167 167 308 308 308 F F F
5 100-119 167 167 167 308 308 308 F F F
6 120-139 167 167 167 308 308 308 F F F
7 140-159 167 167 167 308 308 308 F F F
8 160-179 167 167 167 308 308 308 F F F
9 180-199 167 167 167 308 308 308 F F F
10 200-219 167 167 167 308 308 308 F F F
11 220-239 167 167 167 308 308 308 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 165 165 165 305 305 305 F F F
14 280-299 165 165 165 305 305 305 F F F
15 300-319 167 167 167 308 308 308 F F F
16 320-339 167 167 167 308 308 308 F F F
17 340-359 167 167 167 308 308 308 F F F
18 360-379 167 167 167 308 308 308 F F F
19 380-399 153 153 153 279 279 279 F F F
20 400-419 162 162 162 297 297 297 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 163 163 163 300 300 300 F F F
23 460-479 165 165 165 305 305 305 F F F
24 480-499 164 164 164 303 303 303 F F F
25 500-519 164 164 164 302 302 302 F F F
26 520-539 164 164 164 302 302 302 F F F
27 540-559 164 164 164 302 302 302 F F F
28 560-579 161 161 161 294 294 294 F F F
29 580-599 158 158 158 285 285 285 F F F
Max gen: 167, Max flush: 308
Begin loop 84
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.930391003Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.982236938Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.983317878Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.983362551Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.983377881Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.983392542Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854}
{"msg":"Crucible 7f717f44-b2d7-4fc4-8d0b-3af06888b672 has session id: 3f5ddced-1cdd-4011-b1d0-5f51237fdb2c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984033635Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"Upstairs opts: Upstairs UUID: 7f717f44-b2d7-4fc4-8d0b-3af06888b672, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984080206Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"Crucible stats registered with UUID: 7f717f44-b2d7-4fc4-8d0b-3af06888b672","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984099114Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:49.984115435Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984515017Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"task":"crutest"}
{"msg":"The guest has requested activation with gen:168","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984540781Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854}
{"msg":"7f717f44-b2d7-4fc4-8d0b-3af06888b672 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984609498Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984626048Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"0","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.9846555Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"0","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984670081Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"1","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984684472Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"1","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984697424Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"2","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.98475281Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"2","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984827274Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"io task","client":"2","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984867769Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"io task","client":"0","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984895501Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:64993)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.984922635Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"io task","client":"1","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:35089)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.985169882Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"io task","client":"0","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"ds_connection connected from Ok(127.0.0.1:61523)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.985211916Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"io task","client":"2","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.985698055Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"2","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.985728915Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"0","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.985744496Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"1","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.986361264Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"1","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.986389196Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"0","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.986416579Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"2","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987022324Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.98704644Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"[0]R flush_numbers[0..12]: [308, 300, 300, 284, 308, 308, 308, 308, 308, 308, 308, 308]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987107222Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"downstairs","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"[0]R generation[0..12]: [167, 163, 163, 157, 167, 167, 167, 167, 167, 167, 167, 167]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.98712617Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"downstairs","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987140861Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"downstairs","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"[1]R flush_numbers[0..12]: [308, 300, 300, 284, 308, 308, 308, 308, 308, 308, 308, 308]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987154842Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"downstairs","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"[1]R generation[0..12]: [167, 163, 163, 157, 167, 167, 167, 167, 167, 167, 167, 167]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987168274Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"downstairs","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987181646Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"downstairs","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"[2]R flush_numbers[0..12]: [308, 300, 300, 284, 308, 308, 308, 308, 308, 308, 308, 308]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987194948Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"downstairs","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"[2]R generation[0..12]: [167, 163, 163, 157, 167, 167, 167, 167, 167, 167, 167, 167]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987208329Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"downstairs","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987221931Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"downstairs","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"Max found gen is 168","v":0,"name":"crucible","level":30,"time":"Wait for a query_work_queue command to finish before sending IO
2025-01-27T19:43:49.987237831Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"downstairs","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"Generation requested: 168 >= found:168","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987257259Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"downstairs","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"Next flush: 309","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987277077Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"downstairs","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987291018Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"":"downstairs","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.98730406Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987315683Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"0","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987328385Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"1","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987340867Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"client":"2","":"downstairs","session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987353449Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987365202Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"7f717f44-b2d7-4fc4-8d0b-3af06888b672 is now active with session: 3f5ddced-1cdd-4011-b1d0-5f51237fdb2c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987377355Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.987389107Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854,"session_id":"3f5ddced-1cdd-4011-b1d0-5f51237fdb2c"}
{"msg":"The guest has finished waiting for activation with:168","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:49.98740096Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 154 len 40960 data: 16 11 13 14 15 14 12 17 17 15
02/30 Write block 73 len 32768 data: 17 17 15 12 10 11 10 10
03/30 Write block 225 len 16384 data: 11 12 11 11
04/30 Read block 233 len 4096
05/30 Write block 26 len 16384 data: 15 15 14 14
06/30 Read block 231 len 16384
07/30 Read block 67 len 4096
08/30 Write block 382 len 40960 data: 7 8 8 8 11 10 12 13 13 15
09/30 Write block 75 len 8192 data: 16 13
10/30 Read block 122 len 28672
11/30 Write block 501 len 32768 data: 7 8 8 9 10 11 10 11
12/30 Write block 372 len 32768 data: 15 18 19 17 18 17 16 15
13/30 Read block 344 len 8192
14/30 Read block 348 len 20480
15/30 Read block 518 len 12288
16/30 Write block 537 len 40960 data: 15 13 14 13 12 11 12 10 9 10
17/30 Read block 404 len 16384
18/30 Write block 311 len 36864 data: 11 11 12 15 14 13 12 10 12
19/30 Write block 452 len 36864 data: 10 10 11 15 11 9 8 9 11
20/30 Flush
21/30 Write block 507 len 12288 data: 11 12 9
22/30 Write block 115 len 36864 data: 13 17 20 17 19 17 17 16 12
23/30 Flush
24/30 Write block 333 len 32768 data: 5 6 7 9 10 13 13 13
25/30 Read block 172 len 20480
26/30 Read block 503 len 12288
27/30 Read block 404 len 28672
28/30 Write block 452 len 8192 data: 11 11
29/30 Read block 419 len 12288
30/30 Write block 70 len 28672 data: 17 20 16 18 18 17 14
----------------------------------------------------------------
Crucible gen:168 GIO:true work queues: Upstairs:0 downstairs:7
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1023 Acked Write 8 Done Done Done false
1024 Acked Read 5 Done Done Done false
1025 Acked Read 3 Done Done Done false
1026 Acked Read 7 Done Done Done false
1027 Acked Write 2 Done Done Done false
1028 Acked Read 3 Done Done Done false
1029 Acked Write 7 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 6 6 6 18
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1022 1022 1022
Downstairs last five completed: 1022 1021 1020 1019 1018
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:50.009329411Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1854}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 167 167 167 308 308 308 F F F
1 020-039 168 168 168 309 309 309 F F F
2 040-059 163 163 163 300 300 300 F F F
3 060-079 168 168 168 309 309 309 T T T
4 080-099 168 168 168 309 309 309 F F F
5 100-119 168 168 168 310 310 310 F F F
6 120-139 168 168 168 310 310 310 F F F
7 140-159 168 168 168 309 309 309 F F F
8 160-179 168 168 168 309 309 309 F F F
9 180-199 167 167 167 308 308 308 F F F
10 200-219 167 167 167 308 308 308 F F F
11 220-239 168 168 168 309 309 309 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 165 165 165 305 305 305 F F F
14 280-299 165 165 165 305 305 305 F F F
15 300-319 168 168 168 309 309 309 F F F
16 320-339 167 167 167 308 308 308 T T T
17 340-359 167 167 167 308 308 308 T T T
18 360-379 168 168 168 309 309 309 F F F
19 380-399 168 168 168 309 309 309 F F F
20 400-419 162 162 162 297 297 297 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 168 168 168 309 309 309 T T T
23 460-479 168 168 168 309 309 309 F F F
24 480-499 164 164 164 303 303 303 F F F
25 500-519 168 168 168 310 310 310 F F F
26 520-539 168 168 168 309 309 309 F F F
27 540-559 168 168 168 309 309 309 F F F
28 560-579 161 161 161 294 294 294 F F F
29 580-599 158 158 158 285 285 285 F F F
Max gen: 168, Max flush: 310
A Difference in extent metadata is expected here
On loop 84
Verifying data now
No change after verify
Loop: 84 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 167 167 167 308 308 308 F F F
1 020-039 168 168 168 309 309 309 F F F
2 040-059 163 163 163 300 300 300 F F F
3 060-079 169 169 169 311 311 311 F F F
4 080-099 168 168 168 309 309 309 F F F
5 100-119 168 168 168 310 310 310 F F F
6 120-139 168 168 168 310 310 310 F F F
7 140-159 168 168 168 309 309 309 F F F
8 160-179 168 168 168 309 309 309 F F F
9 180-199 167 167 167 308 308 308 F F F
10 200-219 167 167 167 308 308 308 F F F
11 220-239 168 168 168 309 309 309 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 165 165 165 305 305 305 F F F
14 280-299 165 165 165 305 305 305 F F F
15 300-319 168 168 168 309 309 309 F F F
16 320-339 169 169 169 311 311 311 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 168 168 168 309 309 309 F F F
19 380-399 168 168 168 309 309 309 F F F
20 400-419 162 162 162 297 297 297 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 169 169 169 311 311 311 F F F
23 460-479 168 168 168 309 309 309 F F F
24 480-499 164 164 164 303 303 303 F F F
25 500-519 168 168 168 310 310 310 F F F
26 520-539 168 168 168 309 309 309 F F F
27 540-559 168 168 168 309 309 309 F F F
28 560-579 161 161 161 294 294 294 F F F
29 580-599 158 158 158 285 285 285 F F F
Max gen: 169, Max flush: 311
Begin loop 85
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.256160729Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.301843909Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.302858391Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.302909609Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.302928738Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.302946666Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862}
{"msg":"Crucible 4f7fb46f-2d9a-4d6c-94f5-ab463767ac3d has session id: d42daaf7-413e-4c14-9f2c-be4d4f911b26","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.303559797Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"Upstairs opts: Upstairs UUID: 4f7fb46f-2d9a-4d6c-94f5-ab463767ac3d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.303624097Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"Crucible stats registered with UUID: 4f7fb46f-2d9a-4d6c-94f5-ab463767ac3d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.303662433Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:54.303687078Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.304204747Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"task":"crutest"}
{"msg":"The guest has requested activation with gen:170","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.304237647Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862}
{"msg":"4f7fb46f-2d9a-4d6c-94f5-ab463767ac3d active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.30429673Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.304326572Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"0","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.304350587Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"0","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.304365168Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"1","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.304379399Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"1","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.304392521Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"2","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.304406533Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"2","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.304419385Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"io task","client":"2","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.304443859Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.304471852Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"io task","client":"1","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"ds_connection connected from Ok(127.0.0.1:63502)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.304609947Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"io task","client":"0","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"ds_connection connected from Ok(127.0.0.1:48709)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.30465201Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"io task","client":"2","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"ds_connection connected from Ok(127.0.0.1:40163)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.304680423Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"io task","client":"1","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.305332689Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"2","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.305375562Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"0","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.305965797Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"2","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.306131245Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"0","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.306529758Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.306663915Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.306808115Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"1","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.307306766Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"1","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"[0]R flush_numbers[0..12]: [308, 309, 300, 311, 309, 310, 310, 309, 309, 308, 308, 309]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.3078505Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"downstairs","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"[0]R generation[0..12]: [167, 168, 163, 169, 168, 168, 168, 168, 168, 167, 167, 168]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.307868929Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"downstairs","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.307883779Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"downstairs","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"[1]R flush_numbers[0..12]: [308, 309, 300, 311, 309, 310, 310, 309, 309, 308, 308, 309]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.307897511Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"downstairs","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"[1]R generation[0..12]: [167, 168, 163, 169, 168, 168, 168, 168, 168, 167, 167, 168]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.307916139Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"downstairs","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.30793014Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"downstairs","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"[2]R flush_numbers[0..12]: [308, 309, 300, 311, 309, 310, 310, 309, 309, 308, 308, 309]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.307943572Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"downstairs","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"[2]R generation[0..12]: [167, 168, 163, 169, 168, 168, 168, 168, 168, 167, 167, 168]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.307956914Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"downstairs","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.307970535Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"downstairs","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"Max found gen is 170","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.308001127Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO
d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"Generation requested: 170 >= found:170","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.308020775Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"downstairs","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"Next flush: 312","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.308033267Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"downstairs","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.308045719Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"":"downstairs","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.308058481Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.308070044Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"0","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.308082786Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"1","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.308095248Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"client":"2","":"downstairs","session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.30810922Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.308123101Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"4f7fb46f-2d9a-4d6c-94f5-ab463767ac3d is now active with session: d42daaf7-413e-4c14-9f2c-be4d4f911b26","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.308135703Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.308147276Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862,"session_id":"d42daaf7-413e-4c14-9f2c-be4d4f911b26"}
{"msg":"The guest has finished waiting for activation with:170","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:54.308159109Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 176 len 28672 data: 16 15 16 17 15 18 20
02/30 Read block 353 len 20480
03/30 Write block 476 len 24576 data: 13 15 17 13 15 14
04/30 Read block 244 len 28672
05/30 Write block 319 len 32768 data: 13 11 11 12 15 16 20 19
06/30 Read block 494 len 32768
07/30 Flush
08/30 Write block 327 len 8192 data: 23 18
09/30 Read block 29 len 36864
10/30 Write block 192 len 36864 data: 12 14 12 16 12 13 13 10 9
11/30 Write block 366 len 36864 data: 12 12 16 15 16 16 16 19 20
12/30 Read block 482 len 4096
13/30 Write block 198 len 40960 data: 14 11 10 10 12 12 14 14 13 14
14/30 Read block 303 len 20480
15/30 Read block 348 len 8192
16/30 Write block 522 len 36864 data: 16 14 14 13 11 13 15 17 16
17/30 Write block 378 len 20480 data: 17 16 11 10 8
18/30 Flush
19/30 Write block 587 len 8192 data: 11 12
20/30 Read block 449 len 32768
21/30 Read block 80 len 24576
22/30 Read block 345 len 12288
23/30 Write block 117 len 12288 data: 21 18 20
24/30 Read block 108 len 16384
25/30 Read block 238 len 32768
26/30 Write block 321 len 36864 data: 12 13 16 17 21 20 24 19 15
27/30 Read block 553 len 28672
28/30 Write block 455 len 8192 data: 16 12
29/30 Write block 590 len 24576 data: 14 12 12 12 10 10
30/30 Write block 66 len 20480 data: 16 15 16 14 18
----------------------------------------------------------------
Crucible gen:170 GIO:true work queues: Upstairs:0 downstairs:12
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1018 Acked Write 2 Done Done Done false
1019 Acked Read 8 Done Done Done false
1020 Acked Read 6 Done Done Done false
1021 Acked Read 3 Done Done Done false
1022 Acked Write 3 Done Done Done false
1023 Acked Read 4 Done Done Done false
1024 Acked Read 8 Done Done Done false
1025 Acked Write 9 Done Done Done false
1026 Acked Read 7 Done Done Done false
1027 Acked Write 2 Done Done Done false
1028 Acked Write 6 Sent Sent Sent false
1029 Acked Write 5 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 10 10 10 30
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1017 1017 1017
Downstairs last five completed: 1017 1016 1015 1014 1013
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:54.329036231Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1862}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 167 167 167 308 308 308 F F F
1 020-039 168 168 168 309 309 309 F F F
2 040-059 163 163 163 300 300 300 F F F
3 060-079 169 169 169 311 311 311 T T T
4 080-099 168 168 168 309 309 309 F F F
5 100-119 168 168 168 310 310 310 T T T
6 120-139 168 168 168 310 310 310 F F F
7 140-159 168 168 168 309 309 309 F F F
8 160-179 170 170 170 312 312 312 F F F
9 180-199 170 170 170 313 313 313 F F F
10 200-219 170 170 170 313 313 313 F F F
11 220-239 168 168 168 309 309 309 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 165 165 165 305 305 305 F F F
14 280-299 165 165 165 305 305 305 F F F
15 300-319 170 170 170 312 312 312 F F F
16 320-339 170 170 170 313 313 313 T T T
17 340-359 169 169 169 311 311 311 F F F
18 360-379 170 170 170 313 313 313 F F F
19 380-399 170 170 170 313 313 313 F F F
20 400-419 162 162 162 297 297 297 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 169 169 169 311 311 311 T T T
23 460-479 170 170 170 312 312 312 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 168 168 168 310 310 310 F F F
26 520-539 170 170 170 313 313 313 F F F
27 540-559 168 168 168 309 309 309 F F F
28 560-579 161 161 161 294 294 294 F F F
29 580-599 158 158 158 285 285 285 T T T
Max gen: 170, Max flush: 313
A Difference in extent metadata is expected here
On loop 85
Verifying data now
No change after verify
Loop: 85 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 167 167 167 308 308 308 F F F
1 020-039 168 168 168 309 309 309 F F F
2 040-059 163 163 163 300 300 300 F F F
3 060-079 171 171 171 314 314 314 F F F
4 080-099 168 168 168 309 309 309 F F F
5 100-119 171 171 171 314 314 314 F F F
6 120-139 168 168 168 310 310 310 F F F
7 140-159 168 168 168 309 309 309 F F F
8 160-179 170 170 170 312 312 312 F F F
9 180-199 170 170 170 313 313 313 F F F
10 200-219 170 170 170 313 313 313 F F F
11 220-239 168 168 168 309 309 309 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 165 165 165 305 305 305 F F F
14 280-299 165 165 165 305 305 305 F F F
15 300-319 170 170 170 312 312 312 F F F
16 320-339 171 171 171 314 314 314 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 170 170 170 313 313 313 F F F
19 380-399 170 170 170 313 313 313 F F F
20 400-419 162 162 162 297 297 297 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 171 171 171 314 314 314 F F F
23 460-479 170 170 170 312 312 312 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 168 168 168 310 310 310 F F F
26 520-539 170 170 170 313 313 313 F F F
27 540-559 168 168 168 309 309 309 F F F
28 560-579 161 161 161 294 294 294 F F F
29 580-599 171 171 171 314 314 314 F F F
Max gen: 171, Max flush: 314
Begin loop 86
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.587366519Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:58.631347484Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.632022596Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.664876072Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.666029417Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.666070162Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.666084773Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.666098754Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870}
{"msg":"Crucible c7d1ced2-b351-4667-a364-48f0ddbd414c has session id: 8271f335-e3d8-478c-bbf9-3884f379b0e5","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.66676785Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"Upstairs opts: Upstairs UUID: c7d1ced2-b351-4667-a364-48f0ddbd414c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.666835918Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"Crucible stats registered with UUID: c7d1ced2-b351-4667-a364-48f0ddbd414c","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.666866529Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:58.666887376Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.667370457Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"task":"crutest"}
{"msg":"The guest has requested activation with gen:172","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.667403756Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870}
{"msg":"c7d1ced2-b351-4667-a364-48f0ddbd414c active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.667460361Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.667486465Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"0","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.667535634Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"0","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.667555492Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"1","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.667571622Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"1","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.667592Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"2","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.667615475Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"2","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.667638721Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"io task","client":"2","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.667661867Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"io task","client":"0","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.667684822Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"ds_connection connected from Ok(127.0.0.1:41513)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.667708088Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"io task","client":"2","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"ds_connection connected from Ok(127.0.0.1:44687)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.668065656Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"io task","client":"0","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"ds_connection connected from Ok(127.0.0.1:35430)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.668086444Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"io task","client":"1","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.668588083Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"2","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.668629597Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"1","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.66888393Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"0","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.669420597Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"1","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.669487936Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"2","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.669555923Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"0","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.669994352Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670098358Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"[0]R flush_numbers[0..12]: [308, 309, 300, 314, 309, 314, 310, 309, 312, 313, 313, 309]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670204252Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"downstairs","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"[0]R generation[0..12]: [167, 168, 163, 171, 168, 171, 168, 168, 170, 170, 170, 168]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.6702242Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"downstairs","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670238711Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"downstairs","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"[1]R flush_numbers[0..12]: [308, 309, 300, 314, 309, 314, 310, 309, 312, 313, 313, 309]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670252502Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"downstairs","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"[1]R generation[0..12]: [167, 168, 163, 171, 168, 171, 168, 168, 170, 170, 170, 168]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670266054Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"downstairs","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670279486Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"downstairs","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"[2]R flush_numbers[0..12]: [308, 309, 300, 314, 309, 314, 310, 309, 312, 313, 313, 309]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670292707Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"downstairs","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"[2]R generation[0..12]: [167, 168, 163, 171, 168, 171, 168, 168, 170, 170, 170, 168]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670306299Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"downstairs","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.67032019Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"downstairs","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"Max found gen is 172","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670332742Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"downstairs","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"Generation requested: 172 >= found:172","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670348583Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"downstairs","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
Wait for a query_work_queue command to finish before sending IO
{"msg":"Next flush: 315","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670367261Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"downstairs","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670379753Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"":"downstairs","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670392835Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670404518Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"0","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.67041774Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"1","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670430372Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"client":"2","":"downstairs","session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670443134Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670454887Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"c7d1ced2-b351-4667-a364-48f0ddbd414c is now active with session: 8271f335-e3d8-478c-bbf9-3884f379b0e5","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670467239Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670478752Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870,"session_id":"8271f335-e3d8-478c-bbf9-3884f379b0e5"}
{"msg":"The guest has finished waiting for activation with:172","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:43:58.670491894Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 191 len 36864 data: 13 13 15 13 17 13 14 15 12
02/30 Read block 232 len 36864
03/30 Write block 153 len 32768 data: 18 17 12 14 15 16 15 13
04/30 Write block 58 len 32768 data: 14 12 12 12 15 16 16 17
05/30 Read block 97 len 12288
06/30 Write block 267 len 40960 data: 11 12 15 14 16 15 16 20 20 18
07/30 Read block 376 len 40960
08/30 Read block 522 len 28672
09/30 Read block 547 len 20480
10/30 Read block 120 len 32768
11/30 Read block 147 len 16384
12/30 Write block 56 len 16384 data: 12 12 15 13
13/30 Read block 349 len 28672
14/30 Write block 285 len 28672 data: 11 8 8 8 10 9 13
15/30 Write block 261 len 20480 data: 13 12 12 14 15
16/30 Read block 335 len 12288
17/30 Read block 2 len 20480
18/30 Write block 129 len 40960 data: 12 10 11 9 9 11 10 11 12 14
19/30 Read block 218 len 16384
20/30 Flush
21/30 Write block 409 len 32768 data: 12 13 13 12 12 10 9 12
22/30 Read block 537 len 32768
23/30 Read block 305 len 20480
24/30 Read block 81 len 28672
25/30 Read block 396 len 12288
26/30 Read block 288 len 16384
27/30 Read block 567 len 16384
28/30 Read block 141 len 20480
29/30 Write block 137 len 40960 data: 13 15 15 14 14 16 16 15 16 17
30/30 Write block 593 len 24576 data: 13 11 11 12 7 5
----------------------------------------------------------------
Crucible gen:172 GIO:true work queues: Upstairs:0 downstairs:10
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1020 Acked Write 8 Done Done Done false
1021 Acked Read 8 Done Done Done false
1022 Acked Read 5 Done Done Done false
1023 Acked Read 7 Done Done Done false
1024 Acked Read 3 Done Done Done false
1025 Acked Read 4 Done Done Done false
1026 Acked Read 4 Done Done Done false
1027 Acked Read 5 Done Done Done false
1028 Acked Write 10 Sent Sent Sent false
1029 Acked Write 6 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 8 8 8 24
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1019 1019 1019
Downstairs last five completed: 1019 1018 1017 1016 1015
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:43:58.693798092Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1870}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 167 167 167 308 308 308 F F F
1 020-039 168 168 168 309 309 309 F F F
2 040-059 172 172 172 315 315 315 F F F
3 060-079 172 172 172 315 315 315 F F F
4 080-099 168 168 168 309 309 309 F F F
5 100-119 171 171 171 314 314 314 F F F
6 120-139 172 172 172 315 315 315 T T T
7 140-159 172 172 172 315 315 315 T T T
8 160-179 172 172 172 315 315 315 F F F
9 180-199 172 172 172 315 315 315 F F F
10 200-219 170 170 170 313 313 313 F F F
11 220-239 168 168 168 309 309 309 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 172 172 172 315 315 315 F F F
15 300-319 170 170 170 312 312 312 F F F
16 320-339 171 171 171 314 314 314 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 170 170 170 313 313 313 F F F
19 380-399 170 170 170 313 313 313 F F F
20 400-419 162 162 162 297 297 297 T T T
21 420-439 164 164 164 303 303 303 F F F
22 440-459 171 171 171 314 314 314 F F F
23 460-479 170 170 170 312 312 312 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 168 168 168 310 310 310 F F F
26 520-539 170 170 170 313 313 313 F F F
27 540-559 168 168 168 309 309 309 F F F
28 560-579 161 161 161 294 294 294 F F F
29 580-599 171 171 171 314 314 314 T T T
Max gen: 172, Max flush: 315
A Difference in extent metadata is expected here
On loop 86
Verifying data now
No change after verify
Loop: 86 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 167 167 167 308 308 308 F F F
1 020-039 168 168 168 309 309 309 F F F
2 040-059 172 172 172 315 315 315 F F F
3 060-079 172 172 172 315 315 315 F F F
4 080-099 168 168 168 309 309 309 F F F
5 100-119 171 171 171 314 314 314 F F F
6 120-139 173 173 173 316 316 316 F F F
7 140-159 173 173 173 316 316 316 F F F
8 160-179 172 172 172 315 315 315 F F F
9 180-199 172 172 172 315 315 315 F F F
10 200-219 170 170 170 313 313 313 F F F
11 220-239 168 168 168 309 309 309 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 172 172 172 315 315 315 F F F
15 300-319 170 170 170 312 312 312 F F F
16 320-339 171 171 171 314 314 314 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 170 170 170 313 313 313 F F F
19 380-399 170 170 170 313 313 313 F F F
20 400-419 173 173 173 316 316 316 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 171 171 171 314 314 314 F F F
23 460-479 170 170 170 312 312 312 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 168 168 168 310 310 310 F F F
26 520-539 170 170 170 313 313 313 F F F
27 540-559 168 168 168 309 309 309 F F F
28 560-579 161 161 161 294 294 294 F F F
29 580-599 173 173 173 316 316 316 F F F
Max gen: 173, Max flush: 316
Begin loop 87
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:02.976472584Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:03.021044933Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.021725192Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.05379932Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.057057812Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.057106562Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.057124161Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.05714178Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878}
{"msg":"Crucible 3be08feb-a91b-4fe6-8140-906690fb470f has session id: 5f2e8b43-a356-4a63-811b-9480f0d57dfb","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.057891636Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"Upstairs opts: Upstairs UUID: 3be08feb-a91b-4fe6-8140-906690fb470f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.057953787Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"Crucible stats registered with UUID: 3be08feb-a91b-4fe6-8140-906690fb470f","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.057984399Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:03.058009873Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.058619105Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"task":"crutest"}
{"msg":"The guest has requested activation with gen:174","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.05866084Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878}
{"msg":"3be08feb-a91b-4fe6-8140-906690fb470f active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.058720033Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.058746077Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"0","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.058770741Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"0","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.058785033Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"1","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.058799514Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"1","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.058812795Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"2","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.058826787Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"2","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.058839799Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.05885338Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"io task","client":"0","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.058867102Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"ds_connection connected from Ok(127.0.0.1:34608)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.05903159Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"io task","client":"2","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47830)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.059083598Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"io task","client":"1","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"ds_connection connected from Ok(127.0.0.1:56329)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.059393796Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"io task","client":"0","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.059831495Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"1","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.060007366Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"2","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.060288012Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"0","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.06059787Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"1","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.06062957Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"2","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.060899693Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"0","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061213459Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061299735Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"[0]R flush_numbers[0..12]: [308, 309, 315, 315, 309, 314, 316, 316, 315, 315, 313, 309]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.06142199Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"downstairs","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"[0]R generation[0..12]: [167, 168, 172, 172, 168, 171, 173, 173, 172, 172, 170, 168]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061441957Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"downstairs","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061456618Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"downstairs","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"[1]R flush_numbers[0..12]: [308, 309, 315, 315, 309, 314, 316, 316, 315, 315, 313, 309]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061475107Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"downstairs","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"[1]R generation[0..12]: [167, 168, 172, 172, 168, 171, 173, 173, 172, 172, 170, 168]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061490228Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"downstairs","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061503909Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"downstairs","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"[2]R flush_numbers[0..12]: [308, 309, 315, 315, 309, 314, 316, 316, 315, 315, 313, 309]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061517481Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"downstairs","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"[2]R generation[0..12]: [167, 168, 172, 172, 168, 171, 173, 173, 172, 172, 170, 168]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061531222Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"downstairs","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061544684Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"downstairs","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"Max found gen is 174","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061557416Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"downstairs","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"Generation requested: 174 >= found:174","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061570308Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"downstairs","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
Wait for a query_work_queue command to finish before sending IO
{"msg":"Next flush: 317","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061589416Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"downstairs","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061602258Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"":"downstairs","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.0616155Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061627412Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"0","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061640284Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"1","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061653236Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"client":"2","":"downstairs","session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061668077Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.06168021Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"3be08feb-a91b-4fe6-8140-906690fb470f is now active with session: 5f2e8b43-a356-4a63-811b-9480f0d57dfb","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061692772Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061704395Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878,"session_id":"5f2e8b43-a356-4a63-811b-9480f0d57dfb"}
{"msg":"The guest has finished waiting for activation with:174","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:03.061716467Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 138 len 20480 data: 16 16 15 15 17
02/30 Read block 588 len 32768
03/30 Read block 0 len 20480
04/30 Read block 492 len 12288
05/30 Write block 502 len 40960 data: 9 9 10 11 12 12 13 10 9 11
06/30 Write block 445 len 24576 data: 10 10 10 11 13 10
07/30 Write block 309 len 12288 data: 16 12 12
08/30 Read block 36 len 24576
09/30 Write block 86 len 20480 data: 22 20 19 19 20
10/30 Read block 535 len 40960
11/30 Flush
12/30 Write block 537 len 16384 data: 16 14 15 14
13/30 Write block 21 len 20480 data: 13 14 16 18 15
14/30 Read block 321 len 16384
15/30 Read block 17 len 40960
16/30 Read block 432 len 28672
17/30 Read block 246 len 24576
18/30 Read block 576 len 28672
19/30 Read block 169 len 28672
20/30 Read block 478 len 28672
21/30 Read block 529 len 28672
22/30 Write block 576 len 20480 data: 12 11 9 8 9
23/30 Flush
24/30 Flush
25/30 Write block 71 len 24576 data: 21 17 19 19 18 15
26/30 Write block 510 len 8192 data: 10 12
27/30 Read block 458 len 8192
28/30 Write block 25 len 32768 data: 16 16 16 15 15 14 12 10
29/30 Read block 160 len 16384
30/30 Read block 450 len 32768
----------------------------------------------------------------
Crucible gen:174 GIO:true work queues: Upstairs:0 downstairs:6
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1024 Acked Write 6 Done Done Done false
1025 Acked Write 2 Done Done Done false
1026 Acked Read 2 Done Done Done false
1027 Acked Write 8 Done Done Done false
1028 Acked Read 4 Done Done Done false
1029 Acked Read 8 Sent Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 0 0 1
Done 5 6 6 17
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1023 1023 1023
Downstairs last five completed: 1023 1022 1021 1020 1019
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:03.08331888Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1878}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 167 167 167 308 308 308 F F F
1 020-039 174 174 174 318 318 318 T T T
2 040-059 172 172 172 315 315 315 F F F
3 060-079 172 172 172 315 315 315 T T T
4 080-099 174 174 174 317 317 317 F F F
5 100-119 171 171 171 314 314 314 F F F
6 120-139 174 174 174 317 317 317 F F F
7 140-159 174 174 174 317 317 317 F F F
8 160-179 172 172 172 315 315 315 F F F
9 180-199 172 172 172 315 315 315 F F F
10 200-219 170 170 170 313 313 313 F F F
11 220-239 168 168 168 309 309 309 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 172 172 172 315 315 315 F F F
15 300-319 174 174 174 317 317 317 F F F
16 320-339 171 171 171 314 314 314 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 170 170 170 313 313 313 F F F
19 380-399 170 170 170 313 313 313 F F F
20 400-419 173 173 173 316 316 316 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 174 174 174 317 317 317 F F F
23 460-479 170 170 170 312 312 312 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 174 174 174 317 317 317 T T T
26 520-539 174 174 174 318 318 318 F F F
27 540-559 174 174 174 318 318 318 F F F
28 560-579 174 174 174 318 318 318 F F F
29 580-599 174 174 174 318 318 318 F F F
Max gen: 174, Max flush: 318
A Difference in extent metadata is expected here
On loop 87
Verifying data now
No change after verify
Loop: 87 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 167 167 167 308 308 308 F F F
1 020-039 175 175 175 319 319 319 F F F
2 040-059 172 172 172 315 315 315 F F F
3 060-079 175 175 175 319 319 319 F F F
4 080-099 174 174 174 317 317 317 F F F
5 100-119 171 171 171 314 314 314 F F F
6 120-139 174 174 174 317 317 317 F F F
7 140-159 174 174 174 317 317 317 F F F
8 160-179 172 172 172 315 315 315 F F F
9 180-199 172 172 172 315 315 315 F F F
10 200-219 170 170 170 313 313 313 F F F
11 220-239 168 168 168 309 309 309 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 172 172 172 315 315 315 F F F
15 300-319 174 174 174 317 317 317 F F F
16 320-339 171 171 171 314 314 314 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 170 170 170 313 313 313 F F F
19 380-399 170 170 170 313 313 313 F F F
20 400-419 173 173 173 316 316 316 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 174 174 174 317 317 317 F F F
23 460-479 170 170 170 312 312 312 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 175 175 175 319 319 319 F F F
26 520-539 174 174 174 318 318 318 F F F
27 540-559 174 174 174 318 318 318 F F F
28 560-579 174 174 174 318 318 318 F F F
29 580-599 174 174 174 318 318 318 F F F
Max gen: 175, Max flush: 319
Begin loop 88
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.366688342Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.413001583Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.414137459Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.414188388Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.414212133Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.414230332Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886}
{"msg":"Crucible 7f3d9c13-c3cc-427a-a54b-4696fd1faffd has session id: 5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.41492791Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"Upstairs opts: Upstairs UUID: 7f3d9c13-c3cc-427a-a54b-4696fd1faffd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.414971423Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"Crucible stats registered with UUID: 7f3d9c13-c3cc-427a-a54b-4696fd1faffd","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.414987533Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:07.415000705Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.415536483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"task":"crutest"}
{"msg":"The guest has requested activation with gen:176","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.415565635Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886}
{"msg":"7f3d9c13-c3cc-427a-a54b-4696fd1faffd active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.415623429Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.415668182Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"0","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.415723097Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"0","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.415739717Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"1","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.415753689Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"1","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.415771867Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"2","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.415786218Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"2","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.41580031Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.415820257Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"io task","client":"0","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.415883858Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"io task","client":"1","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"ds_connection connected from Ok(127.0.0.1:48818)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.415920815Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"io task","client":"2","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"ds_connection connected from Ok(127.0.0.1:38815)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.415948948Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"io task","client":"0","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"ds_connection connected from Ok(127.0.0.1:50211)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.416133573Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"io task","client":"1","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.416565836Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"2","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.416601753Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"0","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.417196815Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"0","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.41734945Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"2","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.417803849Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.41789959Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.418322148Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"1","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.418812814Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"1","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"[0]R flush_numbers[0..12]: [308, 319, 315, 319, 317, 314, 317, 317, 315, 315, 313, 309]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419342676Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"downstairs","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"[0]R generation[0..12]: [167, 175, 172, 175, 174, 171, 174, 174, 172, 172, 170, 168]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419362963Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"downstairs","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419378034Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"downstairs","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"[1]R flush_numbers[0..12]: [308, 319, 315, 319, 317, 314, 317, 317, 315, 315, 313, 309]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419392205Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"downstairs","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"[1]R generation[0..12]: [167, 175, 172, 175, 174, 171, 174, 174, 172, 172, 170, 168]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419405627Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"downstairs","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419419169Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"downstairs","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"[2]R flush_numbers[0..12]: [308, 319, 315, 319, 317, 314, 317, 317, 315, 315, 313, 309]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419432571Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"downstairs","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"[2]R generation[0..12]: [167, 175, 172, 175, 174, 171, 174, 174, 172, 172, 170, 168]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419446132Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"downstairs","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419459484Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"downstairs","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"Max found gen is 176","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419472046Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"downstairs","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"Generation requested: 176 >= found:176","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419484828Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"downstairs","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"Next flush: 320","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.41949744Z","hostname"Wait for a query_work_queue command to finish before sending IO
:"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"downstairs","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419522265Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"":"downstairs","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419535836Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419547719Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"0","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419560861Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"1","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419573723Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"client":"2","":"downstairs","session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419586575Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419598478Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"7f3d9c13-c3cc-427a-a54b-4696fd1faffd is now active with session: 5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.41961084Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419622573Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886,"session_id":"5e7b6a4a-ccf2-4f3f-85da-5cd5d3d9e895"}
{"msg":"The guest has finished waiting for activation with:176","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:07.419651095Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 577 len 20480 data: 12 10 9 10 9
02/30 Write block 528 len 16384 data: 16 18 17 15
03/30 Flush
04/30 Write block 134 len 28672 data: 12 11 12 14 17 17 16
05/30 Flush
06/30 Write block 17 len 8192 data: 7 11
07/30 Read block 530 len 28672
08/30 Flush
09/30 Write block 333 len 28672 data: 6 7 8 10 11 14 14
10/30 Write block 97 len 20480 data: 16 18 16 17 18
11/30 Flush
12/30 Write block 231 len 4096 data: 12
13/30 Read block 305 len 12288
14/30 Write block 446 len 28672 data: 11 11 12 14 11 11 12
15/30 Read block 332 len 40960
16/30 Write block 11 len 40960 data: 10 6 5 5 5 5 8 12 11 12
17/30 Read block 70 len 4096
18/30 Read block 20 len 16384
19/30 Write block 281 len 32768 data: 18 15 11 10 12 9 9 9
20/30 Read block 471 len 28672
21/30 Write block 198 len 16384 data: 16 13 11 11
22/30 Read block 184 len 36864
23/30 Read block 254 len 8192
24/30 Flush
25/30 Write block 291 len 32768 data: 14 15 16 15 13 15 14 11
26/30 Write block 390 len 20480 data: 14 16 15 16 14
27/30 Read block 513 len 40960
28/30 Write block 36 len 36864 data: 7 7 8 10 12 14 11 10 10
29/30 Write block 128 len 20480 data: 13 13 11 12 10
30/30 Write block 382 len 32768 data: 9 9 9 9 12 11 13 14
----------------------------------------------------------------
Crucible gen:176 GIO:true work queues: Upstairs:0 downstairs:6
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1024 Acked Write 8 Done Done Done false
1025 Acked Write 5 Done Done Done false
1026 Acked Read 10 Done Done Done false
1027 Acked Write 9 Sent Done Sent false
1028 Acked Write 5 Sent Sent Sent false
1029 Acked Write 8 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 3 2 3 8
Done 3 4 3 10
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1023 1023 1023
Downstairs last five completed: 1023 1022 1021 1020 1019
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:07.4402812Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1886}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 176 176 176 324 324 324 F F F
1 020-039 176 176 176 324 324 324 T T T
2 040-059 172 172 172 315 315 315 T T T
3 060-079 175 175 175 319 319 319 F F F
4 080-099 176 176 176 323 323 323 F F F
5 100-119 176 176 176 323 323 323 F F F
6 120-139 176 176 176 321 321 321 T T T
7 140-159 176 176 176 321 321 321 F F F
8 160-179 172 172 172 315 315 315 F F F
9 180-199 176 176 176 324 324 324 F F F
10 200-219 176 176 176 324 324 324 F F F
11 220-239 176 176 176 324 324 324 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 176 176 176 324 324 324 T T T
15 300-319 174 174 174 317 317 317 F F F
16 320-339 176 176 176 323 323 323 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 170 170 170 313 313 313 F F F
19 380-399 170 170 170 313 313 313 T T T
20 400-419 173 173 173 316 316 316 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 176 176 176 324 324 324 F F F
23 460-479 170 170 170 312 312 312 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 175 175 175 319 319 319 F F F
26 520-539 176 176 176 320 320 320 F F F
27 540-559 174 174 174 318 318 318 F F F
28 560-579 176 176 176 320 320 320 F F F
29 580-599 176 176 176 320 320 320 F F F
Max gen: 176, Max flush: 324
A Difference in extent metadata is expected here
On loop 88
Verifying data now
No change after verify
Loop: 88 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 176 176 176 324 324 324 F F F
1 020-039 177 177 177 325 325 325 F F F
2 040-059 177 177 177 325 325 325 F F F
3 060-079 175 175 175 319 319 319 F F F
4 080-099 176 176 176 323 323 323 F F F
5 100-119 176 176 176 323 323 323 F F F
6 120-139 177 177 177 325 325 325 F F F
7 140-159 176 176 176 321 321 321 F F F
8 160-179 172 172 172 315 315 315 F F F
9 180-199 176 176 176 324 324 324 F F F
10 200-219 176 176 176 324 324 324 F F F
11 220-239 176 176 176 324 324 324 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 177 177 177 325 325 325 F F F
15 300-319 174 174 174 317 317 317 F F F
16 320-339 176 176 176 323 323 323 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 170 170 170 313 313 313 F F F
19 380-399 177 177 177 325 325 325 F F F
20 400-419 173 173 173 316 316 316 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 176 176 176 324 324 324 F F F
23 460-479 170 170 170 312 312 312 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 175 175 175 319 319 319 F F F
26 520-539 176 176 176 320 320 320 F F F
27 540-559 174 174 174 318 318 318 F F F
28 560-579 176 176 176 320 320 320 F F F
29 580-599 176 176 176 320 320 320 F F F
Max gen: 177, Max flush: 325
Begin loop 89
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.707784568Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:11.752358016Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.753043082Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.785674004Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.786755225Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.786797928Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.786814578Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.786832557Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894}
{"msg":"Crucible 11cd3902-1c65-4384-b456-e3e6aa4aa7c8 has session id: 1b64d1c3-c49b-480c-862c-fdee6657c062","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.787405412Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"Upstairs opts: Upstairs UUID: 11cd3902-1c65-4384-b456-e3e6aa4aa7c8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.787462607Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"Crucible stats registered with UUID: 11cd3902-1c65-4384-b456-e3e6aa4aa7c8","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.787490659Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:11.787507988Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.787940341Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"task":"crutest"}
{"msg":"The guest has requested activation with gen:178","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.787972141Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894}
{"msg":"11cd3902-1c65-4384-b456-e3e6aa4aa7c8 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.788046475Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.788068301Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"0","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.788097673Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"0","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.788114803Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"1","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.78818437Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"1","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.788229602Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"2","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.788256595Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"2","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.788278891Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"io task","client":"2","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.788299978Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"io task","client":"0","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.788321735Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"ds_connection connected from Ok(127.0.0.1:54161)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.788497006Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"io task","client":"2","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"ds_connection connected from Ok(127.0.0.1:40820)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.788540229Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"io task","client":"0","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"ds_connection connected from Ok(127.0.0.1:38453)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.788564245Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"io task","client":"1","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.789048834Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"1","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.7890878Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"2","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.789288976Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"0","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.789707457Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"2","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.789818468Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"1","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.79007265Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"0","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790343273Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790433117Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"[0]R flush_numbers[0..12]: [324, 325, 325, 319, 323, 323, 325, 321, 315, 324, 324, 324]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790630945Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"downstairs","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"[0]R generation[0..12]: [176, 177, 177, 175, 176, 176, 177, 176, 172, 176, 176, 176]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790658937Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"downstairs","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790675067Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"downstairs","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"[1]R flush_numbers[0..12]: [324, 325, 325, 319, 323, 323, 325, 321, 315, 324, 324, 324]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790689049Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"downstairs","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"[1]R generation[0..12]: [176, 177, 177, 175, 176, 176, 177, 176, 172, 176, 176, 176]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.79070305Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"downstairs","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790717121Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"downstairs","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"[2]R flush_numbers[0..12]: [324, 325, 325, 319, 323, 323, 325, 321, 315, 324, 324, 324]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.79073494Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"downstairs","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"[2]R generation[0..12]: [176, 177, 177, 175, 176, 176, 177, 176, 172, 176, 176, 176]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790748632Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"downstairs","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790762123Z","hostname":"Wait for a query_work_queue command to finish before sending IO
w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"downstairs","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"Max found gen is 178","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790781182Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"downstairs","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"Generation requested: 178 >= found:178","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790794113Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"downstairs","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"Next flush: 326","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790806526Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"downstairs","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790818948Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"":"downstairs","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.79083158Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790843203Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"0","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790855975Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"1","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790868538Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"client":"2","":"downstairs","session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.79088108Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790892942Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
,"pid":1894,"session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"11cd3902-1c65-4384-b456-e3e6aa4aa7c8 is now active with session: 1b64d1c3-c49b-480c-862c-fdee6657c062","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790909472Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790921055Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894,"session_id":"1b64d1c3-c49b-480c-862c-fdee6657c062"}
{"msg":"The guest has finished waiting for activation with:178","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:11.790934626Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894}
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 197 len 20480 data: 15 17 14 12 12
02/30 Write block 155 len 32768 data: 13 15 16 17 16 14 18 18
03/30 Read block 50 len 36864
04/30 Read block 344 len 4096
05/30 Read block 29 len 32768
06/30 Write block 183 len 8192 data: 19 13
07/30 Write block 173 len 40960 data: 9 10 17 17 16 17 18 16 19 21
08/30 Read block 85 len 24576
09/30 Write block 410 len 24576 data: 14 14 13 13 11 10
10/30 Read block 547 len 24576
11/30 Write block 226 len 8192 data: 13 12
12/30 Write block 321 len 8192 data: 13 14
13/30 Write block 394 len 16384 data: 15 14 13 12
14/30 Write block 466 len 16384 data: 12 12 11 9
15/30 Read block 141 len 36864
16/30 Read block 402 len 24576
17/30 Read block 582 len 8192
18/30 Read block 444 len 8192
19/30 Write block 15 len 28672 data: 6 6 9 13 12 13 14
20/30 Read block 92 len 8192
21/30 Read block 302 len 4096
22/30 Read block 406 len 8192
23/30 Read block 200 len 24576
24/30 Read block 533 len 16384
25/30 Read block 584 len 16384
26/30 Read block 325 len 32768
27/30 Read block 197 len 36864
28/30 Write block 450 len 32768 data: 12 12 13 12 12 17 13 10
29/30 Write block 321 len 16384 data: 14 15 17 18
30/30 Write block 31 len 16384 data: 13 11 10 9
----------------------------------------------------------------
Crucible gen:178 GIO:true work queues: Upstairs:0 downstairs:30
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Write 5 Done Done Done false
1001 Acked Write 8 Done Done Done false
1002 Acked Read 9 Done Done Done false
1003 Acked Read 1 Done Done Done false
1004 Acked Read 8 Done Done Done false
1005 Acked Write 2 Done Done Done false
1006 Acked Write 10 Done Done Done false
1007 Acked Read 6 Done Done Done false
1008 Acked Write 6 Done Done Done false
1009 Acked Read 6 Done Done Done false
1010 Acked Write 2 Done Done Done false
1011 Acked Write 2 Done Done Done false
1012 Acked Write 4 Done Done Done false
1013 Acked Write 4 Done Done Done false
1014 Acked Read 9 Done Done Done false
1015 Acked Read 6 Done Done Done false
1016 Acked Read 2 Done Done Done false
1017 Acked Read 2 Done Done Done false
1018 Acked Write 7 Done Done Done false
1019 Acked Read 2 Done Done Done false
1020 Acked Read 1 Done Done Done false
1021 Acked Read 2 Done Done Done false
1022 Acked Read 6 Done Done Done false
1023 Acked Read 4 Done Done Done false
1024 Acked Read 4 Done Done Done false
1025 Acked Read 8 Done Done Done false
1026 Acked Read 9 Done Done Done false
1027 Acked Write 8 Sent Sent Sent false
1028 Acked Write 4 Sent Sent Sent false
1029 Acked Write 4 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 3 3 3 9
Done 27 27 27 81
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 0 0 0
Downstairs last five completed:
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:11.809708711Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1894}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 176 176 176 324 324 324 T T T
1 020-039 177 177 177 325 325 325 T T T
2 040-059 177 177 177 325 325 325 F F F
3 060-079 175 175 175 319 319 319 F F F
4 080-099 176 176 176 323 323 323 F F F
5 100-119 176 176 176 323 323 323 F F F
6 120-139 177 177 177 325 325 325 F F F
7 140-159 176 176 176 321 321 321 T T T
8 160-179 172 172 172 315 315 315 T T T
9 180-199 176 176 176 324 324 324 T T T
10 200-219 176 176 176 324 324 324 T T T
11 220-239 176 176 176 324 324 324 T T T
12 240-259 167 167 167 308 308 308 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 177 177 177 325 325 325 F F F
15 300-319 174 174 174 317 317 317 F F F
16 320-339 176 176 176 323 323 323 T T T
17 340-359 169 169 169 311 311 311 F F F
18 360-379 170 170 170 313 313 313 F F F
19 380-399 177 177 177 325 325 325 T T T
20 400-419 173 173 173 316 316 316 T T T
21 420-439 164 164 164 303 303 303 F F F
22 440-459 176 176 176 324 324 324 T T T
23 460-479 170 170 170 312 312 312 T T T
24 480-499 170 170 170 312 312 312 F F F
25 500-519 175 175 175 319 319 319 F F F
26 520-539 176 176 176 320 320 320 F F F
27 540-559 174 174 174 318 318 318 F F F
28 560-579 176 176 176 320 320 320 F F F
29 580-599 176 176 176 320 320 320 F F F
Max gen: 177, Max flush: 325
A Difference in extent metadata is expected here
On loop 89
Verifying data now
No change after verify
Loop: 89 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 178 178 178 326 326 326 F F F
1 020-039 178 178 178 326 326 326 F F F
2 040-059 177 177 177 325 325 325 F F F
3 060-079 175 175 175 319 319 319 F F F
4 080-099 176 176 176 323 323 323 F F F
5 100-119 176 176 176 323 323 323 F F F
6 120-139 177 177 177 325 325 325 F F F
7 140-159 178 178 178 326 326 326 F F F
8 160-179 178 178 178 326 326 326 F F F
9 180-199 178 178 178 326 326 326 F F F
10 200-219 178 178 178 326 326 326 F F F
11 220-239 178 178 178 326 326 326 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 177 177 177 325 325 325 F F F
15 300-319 174 174 174 317 317 317 F F F
16 320-339 178 178 178 326 326 326 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 170 170 170 313 313 313 F F F
19 380-399 178 178 178 326 326 326 F F F
20 400-419 178 178 178 326 326 326 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 178 178 178 326 326 326 F F F
23 460-479 178 178 178 326 326 326 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 175 175 175 319 319 319 F F F
26 520-539 176 176 176 320 320 320 F F F
27 540-559 174 174 174 318 318 318 F F F
28 560-579 176 176 176 320 320 320 F F F
29 580-599 176 176 176 320 320 320 F F F
Max gen: 178, Max flush: 326
Begin loop 90
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.142540973Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.189155108Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.190312951Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.190367547Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.190386985Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.190404554Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902}
{"msg":"Crucible f6d8c38a-1824-45dc-8f8e-11c3f3017122 has session id: 8cd7c079-5f42-410e-b5e1-a6f096e7c195","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.193235151Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"Upstairs opts: Upstairs UUID: f6d8c38a-1824-45dc-8f8e-11c3f3017122, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.193272478Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"Crucible stats registered with UUID: f6d8c38a-1824-45dc-8f8e-11c3f3017122","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.193288108Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:16.193300311Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.193722869Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"task":"crutest"}
{"msg":"The guest has requested activation with gen:180","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.193757828Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902}
{"msg":"f6d8c38a-1824-45dc-8f8e-11c3f3017122 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.193817451Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.193855637Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"0","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.19394795Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"0","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.19402867Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"1","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.194097107Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"1","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.194132716Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"2","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.194374256Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"2","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.194400949Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"io task","client":"2","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.194425384Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.194443333Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"io task","client":"0","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"ds_connection connected from Ok(127.0.0.1:34465)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.194472245Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"io task","client":"2","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"ds_connection connected from Ok(127.0.0.1:34674)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.194515348Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"io task","client":"0","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47524)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.194542392Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"io task","client":"1","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.194957814Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"0","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.195007264Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"1","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.195023444Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"2","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.195609981Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"0","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.195761497Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"1","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.19582157Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"2","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.19616033Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196271471Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"[0]R flush_numbers[0..12]: [326, 326, 325, 319, 323, 323, 325, 326, 326, 326, 326, 326]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196394385Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"downstairs","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"[0]R generation[0..12]: [178, 178, 177, 175, 176, 176, 177, 178, 178, 178, 178, 178]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196414882Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"downstairs","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196429693Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"downstairs","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"[1]R flush_numbers[0..12]: [326, 326, 325, 319, 323, 323, 325, 326, 326, 326, 326, 326]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196443585Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"downstairs","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"[1]R generation[0..12]: [178, 178, 177, 175, 176, 176, 177, 178, 178, 178, 178, 178]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196457246Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"downstairs","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196470748Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"downstairs","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"[2]R flush_numbers[0..12]: [326, 326, 325, 319, 323, 323, 325, 326, 326, 326, 326, 326]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196484169Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"downstairs","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"[2]R generation[0..12]: [178, 178, 177, 175, 176, 176, 177, 178, 178, 178, 178, 178]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196497401Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"downstairs","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196510753Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"downstairs","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"Max found gen is 179","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196523375Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"downstairs","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"Generation requested: 180 >= found:179","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196536217Z","hostnameWait for a query_work_queue command to finish before sending IO
":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"downstairs","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"Next flush: 327","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196558513Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"downstairs","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196571195Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"":"downstairs","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196584027Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.19659581Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"0","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196608962Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"1","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196621874Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"client":"2","":"downstairs","session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196634896Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196646799Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"f6d8c38a-1824-45dc-8f8e-11c3f3017122 is now active with session: 8cd7c079-5f42-410e-b5e1-a6f096e7c195","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196659401Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196671173Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902,"session_id":"8cd7c079-5f42-410e-b5e1-a6f096e7c195"}
{"msg":"The guest has finished waiting for activation with:180","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:16.196682936Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 210 len 24576 data: 12 14 13 12 11 12
02/30 Write block 70 len 36864 data: 19 22 18 20 20 19 16 11 12
03/30 Flush
04/30 Write block 101 len 4096 data: 19
05/30 Write block 443 len 28672 data: 10 8 11 12 12 13 15
06/30 Flush
07/30 Write block 377 len 8192 data: 18 18
08/30 Read block 17 len 24576
09/30 Read block 54 len 36864
10/30 Write block 212 len 36864 data: 14 13 12 13 10 10 11 10 8
11/30 Read block 572 len 40960
12/30 Read block 345 len 36864
13/30 Read block 303 len 28672
14/30 Write block 283 len 28672 data: 12 11 13 10 10 10 11
15/30 Read block 192 len 20480
16/30 Read block 443 len 16384
17/30 Read block 545 len 20480
18/30 Write block 448 len 16384 data: 14 16 13 13
19/30 Read block 86 len 16384
20/30 Write block 51 len 16384 data: 10 10 10 11
21/30 Write block 541 len 40960 data: 13 12 13 11 10 11 9 11 14 13
22/30 Read block 229 len 12288
23/30 Flush
24/30 Write block 471 len 8192 data: 11 10
25/30 Write block 388 len 8192 data: 14 15
26/30 Read block 461 len 24576
27/30 Write block 322 len 36864 data: 16 18 19 22 21 25 20 16 13
28/30 Read block 510 len 28672
29/30 Read block 246 len 32768
30/30 Write block 541 len 28672 data: 14 13 14 12 11 12 10
----------------------------------------------------------------
Crucible gen:180 GIO:true work queues: Upstairs:0 downstairs:7
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1023 Acked Write 2 Done Done Done false
1024 Acked Write 2 Done Done Done false
1025 Acked Read 6 Done Done Done false
1026 Acked Write 9 Done Done Done false
1027 Acked Read 7 Done Done Done false
1028 Acked Read 8 Done Done Done false
1029 Acked Write 7 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 6 6 6 18
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1022 1022 1022
Downstairs last five completed: 1022 1021 1020 1019 1018
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:16.218017715Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1902}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 178 178 178 326 326 326 F F F
1 020-039 178 178 178 326 326 326 F F F
2 040-059 180 180 180 329 329 329 F F F
3 060-079 180 180 180 327 327 327 F F F
4 080-099 176 176 176 323 323 323 F F F
5 100-119 180 180 180 328 328 328 F F F
6 120-139 177 177 177 325 325 325 F F F
7 140-159 178 178 178 326 326 326 F F F
8 160-179 178 178 178 326 326 326 F F F
9 180-199 178 178 178 326 326 326 F F F
10 200-219 180 180 180 329 329 329 F F F
11 220-239 180 180 180 329 329 329 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 180 180 180 329 329 329 F F F
15 300-319 174 174 174 317 317 317 F F F
16 320-339 178 178 178 326 326 326 T T T
17 340-359 169 169 169 311 311 311 F F F
18 360-379 180 180 180 329 329 329 F F F
19 380-399 178 178 178 326 326 326 T T T
20 400-419 178 178 178 326 326 326 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 180 180 180 329 329 329 F F F
23 460-479 178 178 178 326 326 326 T T T
24 480-499 170 170 170 312 312 312 F F F
25 500-519 175 175 175 319 319 319 F F F
26 520-539 176 176 176 320 320 320 F F F
27 540-559 180 180 180 329 329 329 T T T
28 560-579 176 176 176 320 320 320 F F F
29 580-599 176 176 176 320 320 320 F F F
Max gen: 180, Max flush: 329
A Difference in extent metadata is expected here
On loop 90
Verifying data now
No change after verify
Loop: 90 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 178 178 178 326 326 326 F F F
1 020-039 178 178 178 326 326 326 F F F
2 040-059 180 180 180 329 329 329 F F F
3 060-079 180 180 180 327 327 327 F F F
4 080-099 176 176 176 323 323 323 F F F
5 100-119 180 180 180 328 328 328 F F F
6 120-139 177 177 177 325 325 325 F F F
7 140-159 178 178 178 326 326 326 F F F
8 160-179 178 178 178 326 326 326 F F F
9 180-199 178 178 178 326 326 326 F F F
10 200-219 180 180 180 329 329 329 F F F
11 220-239 180 180 180 329 329 329 F F F
12 240-259 167 167 167 308 308 308 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 180 180 180 329 329 329 F F F
15 300-319 174 174 174 317 317 317 F F F
16 320-339 181 181 181 330 330 330 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 180 180 180 329 329 329 F F F
19 380-399 181 181 181 330 330 330 F F F
20 400-419 178 178 178 326 326 326 F F F
21 420-439 164 164 164 303 303 303 F F F
22 440-459 180 180 180 329 329 329 F F F
23 460-479 181 181 181 330 330 330 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 175 175 175 319 319 319 F F F
26 520-539 176 176 176 320 320 320 F F F
27 540-559 181 181 181 330 330 330 F F F
28 560-579 176 176 176 320 320 320 F F F
29 580-599 176 176 176 320 320 320 F F F
Max gen: 181, Max flush: 330
Begin loop 91
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.474403605Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.52114718Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.522248088Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.522307192Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.522329827Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.522351674Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910}
{"msg":"Crucible 645aa026-06af-464e-99e5-b3126d92a6b0 has session id: b1334744-3937-432b-af31-7b2d93488fc8","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523047413Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"Upstairs opts: Upstairs UUID: 645aa026-06af-464e-99e5-b3126d92a6b0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523083241Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"Crucible stats registered with UUID: 645aa026-06af-464e-99e5-b3126d92a6b0","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523099241Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:20.523113082Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523570319Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"task":"crutest"}
{"msg":"The guest has requested activation with gen:182","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523595544Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910}
{"msg":"645aa026-06af-464e-99e5-b3126d92a6b0 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523655397Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523673625Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"0","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523696991Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"0","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523719877Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"1","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523748689Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"1","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523777441Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"2","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523805574Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"2","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523829349Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"io task","client":"2","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523861549Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"io task","client":"0","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523890951Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"ds_connection connected from Ok(127.0.0.1:61836)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523920053Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"io task","client":"2","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"ds_connection connected from Ok(127.0.0.1:42127)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523950964Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"io task","client":"0","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"ds_connection connected from Ok(127.0.0.1:48834)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.523980276Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"io task","client":"1","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.524728912Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"2","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.524889522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"1","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.524935614Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"0","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.525534333Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"2","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.525575967Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"0","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.525754387Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"1","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526140738Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526241875Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"[0]R flush_numbers[0..12]: [326, 326, 329, 327, 323, 328, 325, 326, 326, 326, 329, 329]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526367127Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"downstairs","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"[0]R generation[0..12]: [178, 178, 180, 180, 176, 180, 177, 178, 178, 178, 180, 180]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526388934Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"downstairs","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526403915Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"downstairs","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"[1]R flush_numbers[0..12]: [326, 326, 329, 327, 323, 328, 325, 326, 326, 326, 329, 329]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526417896Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"downstairs","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"[1]R generation[0..12]: [178, 178, 180, 180, 176, 180, 177, 178, 178, 178, 180, 180]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526431578Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"downstairs","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526445509Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"downstairs","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"[2]R flush_numbers[0..12]: [326, 326, 329, 327, 323, 328, 325, 326, 326, 326, 329, 329]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526464237Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"downstairs","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"[2]R generation[0..12]: [178, 178, 180, 180, 176, 180, 177, 178, 178, 178, 180, 180]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526478658Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"downstairs","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526493069Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"downstairs","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"Max found gen is 182","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
,"time":"2025-01-27T19:44:20.526506071Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"downstairs","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"Generation requested: 182 >= found:182","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.52652434Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"downstairs","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"Next flush: 331","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526537782Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"downstairs","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526550494Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"":"downstairs","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526563716Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526575598Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"0","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.52658892Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"1","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526601822Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"client":"2","":"downstairs","session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526614784Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526626747Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"645aa026-06af-464e-99e5-b3126d92a6b0 is now active with session: b1334744-3937-432b-af31-7b2d93488fc8","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526639469Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526651182Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910,"session_id":"b1334744-3937-432b-af31-7b2d93488fc8"}
{"msg":"The guest has finished waiting for activation with:182","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:20.526665253Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 520 len 28672 data: 12 14 17 15 15 14 12
02/30 Write block 569 len 28672 data: 10 12 12 13 14 13 15
03/30 Read block 568 len 8192
04/30 Write block 419 len 20480 data: 16 17 15 16 17
05/30 Flush
06/30 Write block 431 len 20480 data: 12 15 15 17 20
07/30 Write block 228 len 8192 data: 12 11
08/30 Read block 372 len 36864
09/30 Write block 235 len 8192 data: 9 6
10/30 Flush
11/30 Write block 239 len 24576 data: 4 7 10 13 15 14
12/30 Flush
13/30 Write block 70 len 28672 data: 20 23 19 21 21 20 17
14/30 Read block 205 len 40960
15/30 Read block 209 len 28672
16/30 Read block 274 len 24576
17/30 Flush
18/30 Write block 553 len 4096 data: 11
19/30 Read block 185 len 40960
20/30 Write block 402 len 28672 data: 15 12 14 15 13 12 11
21/30 Read block 85 len 24576
22/30 Read block 174 len 4096
23/30 Read block 390 len 8192
24/30 Read block 8 len 12288
25/30 Flush
26/30 Write block 121 len 40960 data: 18 17 13 14 9 11 13 14 14 12
27/30 Read block 183 len 24576
28/30 Write block 96 len 24576 data: 17 17 19 17 18 20
29/30 Read block 414 len 24576
30/30 Read block 383 len 40960
----------------------------------------------------------------
Crucible gen:182 GIO:true work queues: Upstairs:0 downstairs:5
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1025 Acked Write 10 Done Done Done false
1026 Acked Read 6 Done Done Done false
1027 Acked Write 6 Done Done Done false
1028 Acked Read 6 Done Done Done false
1029 Acked Read 10 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 0 1
Done 5 4 5 14
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1024 1024 1024
Downstairs last five completed: 1024 1023 1022 1021 1020
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:20.547794279Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1910}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 178 178 178 326 326 326 F F F
1 020-039 178 178 178 326 326 326 F F F
2 040-059 180 180 180 329 329 329 F F F
3 060-079 182 182 182 334 334 334 F F F
4 080-099 176 176 176 323 323 323 T T T
5 100-119 180 180 180 328 328 328 T T T
6 120-139 177 177 177 325 325 325 T T T
7 140-159 178 178 178 326 326 326 F F F
8 160-179 178 178 178 326 326 326 F F F
9 180-199 178 178 178 326 326 326 F F F
10 200-219 180 180 180 329 329 329 F F F
11 220-239 182 182 182 333 333 333 F F F
12 240-259 182 182 182 333 333 333 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 180 180 180 329 329 329 F F F
15 300-319 174 174 174 317 317 317 F F F
16 320-339 181 181 181 330 330 330 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 180 180 180 329 329 329 F F F
19 380-399 181 181 181 330 330 330 F F F
20 400-419 182 182 182 335 335 335 F F F
21 420-439 182 182 182 332 332 332 F F F
22 440-459 180 180 180 329 329 329 F F F
23 460-479 181 181 181 330 330 330 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 175 175 175 319 319 319 F F F
26 520-539 182 182 182 331 331 331 F F F
27 540-559 182 182 182 335 335 335 F F F
28 560-579 182 182 182 331 331 331 F F F
29 580-599 176 176 176 320 320 320 F F F
Max gen: 182, Max flush: 335
A Difference in extent metadata is expected here
On loop 91
Verifying data now
No change after verify
Loop: 91 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 178 178 178 326 326 326 F F F
1 020-039 178 178 178 326 326 326 F F F
2 040-059 180 180 180 329 329 329 F F F
3 060-079 182 182 182 334 334 334 F F F
4 080-099 183 183 183 336 336 336 F F F
5 100-119 183 183 183 336 336 336 F F F
6 120-139 183 183 183 336 336 336 F F F
7 140-159 178 178 178 326 326 326 F F F
8 160-179 178 178 178 326 326 326 F F F
9 180-199 178 178 178 326 326 326 F F F
10 200-219 180 180 180 329 329 329 F F F
11 220-239 182 182 182 333 333 333 F F F
12 240-259 182 182 182 333 333 333 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 180 180 180 329 329 329 F F F
15 300-319 174 174 174 317 317 317 F F F
16 320-339 181 181 181 330 330 330 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 180 180 180 329 329 329 F F F
19 380-399 181 181 181 330 330 330 F F F
20 400-419 182 182 182 335 335 335 F F F
21 420-439 182 182 182 332 332 332 F F F
22 440-459 180 180 180 329 329 329 F F F
23 460-479 181 181 181 330 330 330 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 175 175 175 319 319 319 F F F
26 520-539 182 182 182 331 331 331 F F F
27 540-559 182 182 182 335 335 335 F F F
28 560-579 182 182 182 331 331 331 F F F
29 580-599 176 176 176 320 320 320 F F F
Max gen: 183, Max flush: 336
Begin loop 92
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.79600557Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.842128389Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.843110611Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.843162199Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.843179908Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.843197377Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918}
{"msg":"Crucible 9c4d8617-3efb-4266-affd-9b4b69783639 has session id: 4b3b5994-6fc3-4f1a-a934-47664384ff1d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.843865114Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"Upstairs opts: Upstairs UUID: 9c4d8617-3efb-4266-affd-9b4b69783639, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.84391876Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"Crucible stats registered with UUID: 9c4d8617-3efb-4266-affd-9b4b69783639","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.843946463Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:24.843971918Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.844541245Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"task":"crutest"}
{"msg":"The guest has requested activation with gen:184","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.844576533Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918}
{"msg":"9c4d8617-3efb-4266-affd-9b4b69783639 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.844633838Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.844658083Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"0","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.844683877Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"0","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.844697948Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"1","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.844712289Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"1","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.844725871Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"2","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.844743989Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"2","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.844767345Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.84479237Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"io task","client":"0","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.844815445Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"io task","client":"1","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:43362)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.844865574Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"io task","client":"1","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47340)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.844896245Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"io task","client":"2","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:45002)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.84492098Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"io task","client":"0","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.845538417Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"0","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.845574845Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"1","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.845773262Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"2","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.846168028Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"1","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.846249037Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"0","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.846510456Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"2","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.84677817Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.846859689Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"[0]R flush_numbers[0..12]: [326, 326, 329, 334, 336, 336, 336, 326, 326, 326, 329, 333]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847026366Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"downstairs","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"[0]R generation[0..12]: [178, 178, 180, 182, 183, 183, 183, 178, 178, 178, 180, 182]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847047023Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"downstairs","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847065912Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"downstairs","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"[1]R flush_numbers[0..12]: [326, 326, 329, 334, 336, 336, 336, 326, 326, 326, 329, 333]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847079803Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"downstairs","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"[1]R generation[0..12]: [178, 178, 180, 182, 183, 183, 183, 178, 178, 178, 180, 182]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847093325Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"downstairs","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847106646Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"downstairs","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"[2]R flush_numbers[0..12]: [326, 326, 329, 334, 336, 336, 336, 326, 326, 326, 329, 333]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847119858Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"downstairs","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"[2]R generation[0..12]: [178, 178, 180, 182, 183, 183, 183, 178, 178, 178, 180, 182]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.84713349Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"downstairs","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847147161Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"downstairs","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"Max found gen is 184","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847159524Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"downstairs","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
Wait for a query_work_queue command to finish before sending IO
{"msg":"Generation requested: 184 >= found:184","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847178822Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"downstairs","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"Next flush: 337","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847191754Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"downstairs","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847204366Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"":"downstairs","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847217298Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847228801Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"0","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847241493Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"1","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847254295Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"client":"2","":"downstairs","session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847268926Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847280918Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"9c4d8617-3efb-4266-affd-9b4b69783639 is now active with session: 4b3b5994-6fc3-4f1a-a934-47664384ff1d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847293491Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847305004Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918,"session_id":"4b3b5994-6fc3-4f1a-a934-47664384ff1d"}
{"msg":"The guest has finished waiting for activation with:184","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:24.847316756Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 400 len 4096 data: 14
02/30 Read block 57 len 8192
03/30 Flush
04/30 Write block 89 len 24576 data: 20 21 17 20 24 19
05/30 Write block 99 len 4096 data: 18
06/30 Read block 517 len 40960
07/30 Read block 109 len 8192
08/30 Read block 85 len 4096
09/30 Read block 30 len 20480
10/30 Read block 108 len 4096
11/30 Read block 311 len 36864
12/30 Write block 246 len 36864 data: 18 19 18 15 12 10 10 12 12
13/30 Write block 580 len 16384 data: 11 10 7 6
14/30 Read block 206 len 16384
15/30 Write block 86 len 24576 data: 23 21 20 21 22 18
16/30 Flush
17/30 Write block 71 len 32768 data: 24 20 22 22 21 18 12 13
18/30 Flush
19/30 Write block 282 len 4096 data: 16
20/30 Write block 59 len 36864 data: 14 13 13 16 17 17 18 17 16
21/30 Read block 432 len 40960
22/30 Read block 456 len 8192
23/30 Read block 190 len 12288
24/30 Flush
25/30 Write block 64 len 32768 data: 18 19 18 17 17 15 21 25
26/30 Read block 67 len 16384
27/30 Read block 585 len 36864
28/30 Write block 302 len 28672 data: 12 14 13 14 16 16 17
29/30 Read block 474 len 20480
30/30 Read block 190 len 4096
----------------------------------------------------------------
Crucible gen:184 GIO:true work queues: Upstairs:0 downstairs:6
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1024 Acked Write 8 Done Done Done false
1025 Acked Read 4 Done Done Done false
1026 Acked Read 9 Done Done Done false
1027 Acked Write 7 Done Done Done false
1028 Acked Read 5 Done Done Done false
1029 Acked Read 1 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 6 6 6 18
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1023 1023 1023
Downstairs last five completed: 1023 1022 1021 1020 1019
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:24.868119234Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1918}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 178 178 178 326 326 326 F F F
1 020-039 178 178 178 326 326 326 F F F
2 040-059 184 184 184 340 340 340 F F F
3 060-079 184 184 184 340 340 340 T T T
4 080-099 184 184 184 338 338 338 F F F
5 100-119 183 183 183 336 336 336 F F F
6 120-139 183 183 183 336 336 336 F F F
7 140-159 178 178 178 326 326 326 F F F
8 160-179 178 178 178 326 326 326 F F F
9 180-199 178 178 178 326 326 326 F F F
10 200-219 180 180 180 329 329 329 F F F
11 220-239 182 182 182 333 333 333 F F F
12 240-259 184 184 184 338 338 338 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 184 184 184 340 340 340 F F F
15 300-319 174 174 174 317 317 317 T T T
16 320-339 181 181 181 330 330 330 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 180 180 180 329 329 329 F F F
19 380-399 181 181 181 330 330 330 F F F
20 400-419 184 184 184 337 337 337 F F F
21 420-439 182 182 182 332 332 332 F F F
22 440-459 180 180 180 329 329 329 F F F
23 460-479 181 181 181 330 330 330 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 175 175 175 319 319 319 F F F
26 520-539 182 182 182 331 331 331 F F F
27 540-559 182 182 182 335 335 335 F F F
28 560-579 182 182 182 331 331 331 F F F
29 580-599 184 184 184 338 338 338 F F F
Max gen: 184, Max flush: 340
A Difference in extent metadata is expected here
On loop 92
Verifying data now
No change after verify
Loop: 92 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 178 178 178 326 326 326 F F F
1 020-039 178 178 178 326 326 326 F F F
2 040-059 184 184 184 340 340 340 F F F
3 060-079 185 185 185 341 341 341 F F F
4 080-099 184 184 184 338 338 338 F F F
5 100-119 183 183 183 336 336 336 F F F
6 120-139 183 183 183 336 336 336 F F F
7 140-159 178 178 178 326 326 326 F F F
8 160-179 178 178 178 326 326 326 F F F
9 180-199 178 178 178 326 326 326 F F F
10 200-219 180 180 180 329 329 329 F F F
11 220-239 182 182 182 333 333 333 F F F
12 240-259 184 184 184 338 338 338 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 184 184 184 340 340 340 F F F
15 300-319 185 185 185 341 341 341 F F F
16 320-339 181 181 181 330 330 330 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 180 180 180 329 329 329 F F F
19 380-399 181 181 181 330 330 330 F F F
20 400-419 184 184 184 337 337 337 F F F
21 420-439 182 182 182 332 332 332 F F F
22 440-459 180 180 180 329 329 329 F F F
23 460-479 181 181 181 330 330 330 F F F
24 480-499 170 170 170 312 312 312 F F F
25 500-519 175 175 175 319 319 319 F F F
26 520-539 182 182 182 331 331 331 F F F
27 540-559 182 182 182 335 335 335 F F F
28 560-579 182 182 182 331 331 331 F F F
29 580-599 184 184 184 338 338 338 F F F
Max gen: 185, Max flush: 341
Begin loop 93
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.105886706Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.152713718Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.153658344Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.153706843Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.153721614Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.153737415Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926}
{"msg":"Crucible 5b96ddfc-f2cd-45af-b7cb-bd2aa63ef5a3 has session id: 03f3d603-7bea-45ca-a156-d269c495109d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.154368324Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"Upstairs opts: Upstairs UUID: 5b96ddfc-f2cd-45af-b7cb-bd2aa63ef5a3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.154415624Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"Crucible stats registered with UUID: 5b96ddfc-f2cd-45af-b7cb-bd2aa63ef5a3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.154440629Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:29.154464774Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.154954271Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"task":"crutest"}
{"msg":"The guest has requested activation with gen:186","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.154977217Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926}
{"msg":"5b96ddfc-f2cd-45af-b7cb-bd2aa63ef5a3 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.155050681Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.15506936Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"0","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.155095214Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"0","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.155109235Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"1","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.155123526Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"1","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.155139886Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"2","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.155154087Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"2","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.155216738Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"io task","client":"2","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.155250158Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.15531159Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"io task","client":"1","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49885)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.155350955Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"io task","client":"2","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:53410)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.155380237Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"io task","client":"0","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:65422)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.155412857Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"io task","client":"1","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.156169139Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"1","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.156224724Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"0","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.15624781Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"2","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157014815Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"1","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157045456Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"0","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157060477Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"2","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157646314Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.15773298Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"[0]R flush_numbers[0..12]: [326, 326, 340, 341, 338, 336, 336, 326, 326, 326, 329, 333]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157866378Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"downstairs","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"[0]R generation[0..12]: [178, 178, 184, 185, 184, 183, 183, 178, 178, 178, 180, 182]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157888594Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"downstairs","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157903435Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"downstairs","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"[1]R flush_numbers[0..12]: [326, 326, 340, 341, 338, 336, 336, 326, 326, 326, 329, 333]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157917286Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"downstairs","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"[1]R generation[0..12]: [178, 178, 184, 185, 184, 183, 183, 178, 178, 178, 180, 182]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157930738Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"downstairs","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157944339Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"downstairs","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"[2]R flush_numbers[0..12]: [326, 326, 340, 341, 338, 336, 336, 326, 326, 326, 329, 333]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157957421Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"downstairs","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"[2]R generation[0..12]: [178, 178, 184, 185, 184, 183, 183, 178, 178, 178, 180, 182]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157970753Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"downstairs","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157984394Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"downstairs","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"Max found gen is 186","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.157996947Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"downstairs"Wait for a query_work_queue command to finish before sending IO
,"":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"Generation requested: 186 >= found:186","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.158016785Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"downstairs","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"Next flush: 342","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.158029167Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"downstairs","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.158045517Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"":"downstairs","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.158058588Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.158070611Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"0","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.158083803Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"1","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.158096295Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"client":"2","":"downstairs","session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.158108877Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.15812072Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"5b96ddfc-f2cd-45af-b7cb-bd2aa63ef5a3 is now active with session: 03f3d603-7bea-45ca-a156-d269c495109d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.158133332Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.158144815Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926,"session_id":"03f3d603-7bea-45ca-a156-d269c495109d"}
{"msg":"The guest has finished waiting for activation with:186","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:29.158156688Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 594 len 16384 data: 12 12 13 8
02/30 Write block 450 len 8192 data: 14 14
03/30 Write block 567 len 4096 data: 10
04/30 Write block 3 len 4096 data: 11
05/30 Write block 236 len 24576 data: 7 6 5 5 8 11
06/30 Read block 420 len 20480
07/30 Read block 118 len 20480
08/30 Write block 422 len 36864 data: 17 18 14 13 11 11 8 10 12
09/30 Write block 483 len 40960 data: 13 14 12 12 16 16 13 15 18 21
10/30 Write block 294 len 12288 data: 16 14 16
11/30 Write block 60 len 28672 data: 14 14 17 18 19 20 19
12/30 Write block 506 len 8192 data: 13 13
13/30 Write block 106 len 8192 data: 17 13
14/30 Read block 284 len 16384
15/30 Flush
16/30 Write block 67 len 28672 data: 18 18 16 22 26 21 23
17/30 Flush
18/30 Write block 494 len 36864 data: 14 19 19 15 15 14 9 8 10
19/30 Write block 575 len 36864 data: 16 13 13 11 10 12 11 8 7
20/30 Write block 462 len 4096 data: 16
21/30 Read block 154 len 4096
22/30 Read block 61 len 8192
23/30 Read block 381 len 16384
24/30 Read block 317 len 40960
25/30 Read block 21 len 32768
26/30 Write block 91 len 28672 data: 19 21 25 20 19 18 18
27/30 Read block 237 len 28672
28/30 Write block 97 len 28672 data: 19 20 19 19 21 19 20
29/30 Read block 94 len 32768
30/30 Read block 261 len 16384
----------------------------------------------------------------
Crucible gen:186 GIO:true work queues: Upstairs:0 downstairs:13
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1017 Acked Write 9 Done Done Done false
1018 Acked Write 9 Done Done Done false
1019 Acked Write 1 Done Done Done false
1020 Acked Read 1 Done Done Done false
1021 Acked Read 2 Done Done Done false
1022 Acked Read 4 Done Done Done false
1023 Acked Read 10 Done Done Done false
1024 Acked Read 8 Done Done Done false
1025 Acked Write 7 Done Done Done false
1026 Acked Read 7 Done Done Done false
1027 Acked Write 7 Done Done Done false
1028 Acked Read 8 Done Done Done false
1029 Acked Read 4 Done Sent Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 1 0 1
Done 13 12 13 38
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1016 1016 1016
Downstairs last five completed: 1016 1015 1014 1013 1012
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:29.176912044Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1926}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 186 186 186 342 342 342 F F F
1 020-039 178 178 178 326 326 326 F F F
2 040-059 184 184 184 340 340 340 F F F
3 060-079 186 186 186 343 343 343 F F F
4 080-099 184 184 184 338 338 338 T T T
5 100-119 186 186 186 342 342 342 T T T
6 120-139 183 183 183 336 336 336 F F F
7 140-159 178 178 178 326 326 326 F F F
8 160-179 178 178 178 326 326 326 F F F
9 180-199 178 178 178 326 326 326 F F F
10 200-219 180 180 180 329 329 329 F F F
11 220-239 186 186 186 342 342 342 F F F
12 240-259 186 186 186 342 342 342 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 186 186 186 342 342 342 F F F
15 300-319 185 185 185 341 341 341 F F F
16 320-339 181 181 181 330 330 330 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 180 180 180 329 329 329 F F F
19 380-399 181 181 181 330 330 330 F F F
20 400-419 184 184 184 337 337 337 F F F
21 420-439 186 186 186 342 342 342 F F F
22 440-459 186 186 186 342 342 342 F F F
23 460-479 181 181 181 330 330 330 T T T
24 480-499 186 186 186 342 342 342 T T T
25 500-519 186 186 186 342 342 342 T T T
26 520-539 182 182 182 331 331 331 F F F
27 540-559 182 182 182 335 335 335 F F F
28 560-579 186 186 186 342 342 342 T T T
29 580-599 186 186 186 342 342 342 T T T
Max gen: 186, Max flush: 343
A Difference in extent metadata is expected here
On loop 93
Verifying data now
No change after verify
Loop: 93 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 186 186 186 342 342 342 F F F
1 020-039 178 178 178 326 326 326 F F F
2 040-059 184 184 184 340 340 340 F F F
3 060-079 186 186 186 343 343 343 F F F
4 080-099 187 187 187 344 344 344 F F F
5 100-119 187 187 187 344 344 344 F F F
6 120-139 183 183 183 336 336 336 F F F
7 140-159 178 178 178 326 326 326 F F F
8 160-179 178 178 178 326 326 326 F F F
9 180-199 178 178 178 326 326 326 F F F
10 200-219 180 180 180 329 329 329 F F F
11 220-239 186 186 186 342 342 342 F F F
12 240-259 186 186 186 342 342 342 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 186 186 186 342 342 342 F F F
15 300-319 185 185 185 341 341 341 F F F
16 320-339 181 181 181 330 330 330 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 180 180 180 329 329 329 F F F
19 380-399 181 181 181 330 330 330 F F F
20 400-419 184 184 184 337 337 337 F F F
21 420-439 186 186 186 342 342 342 F F F
22 440-459 186 186 186 342 342 342 F F F
23 460-479 187 187 187 344 344 344 F F F
24 480-499 187 187 187 344 344 344 F F F
25 500-519 187 187 187 344 344 344 F F F
26 520-539 182 182 182 331 331 331 F F F
27 540-559 182 182 182 335 335 335 F F F
28 560-579 187 187 187 344 344 344 F F F
29 580-599 187 187 187 344 344 344 F F F
Max gen: 187, Max flush: 344
Begin loop 94
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.442387455Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.487817863Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.488953969Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.489004798Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.489023007Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.489038847Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934}
{"msg":"Crucible 66b67456-fe07-431b-83a3-40c081601672 has session id: 2014967c-3f69-46b9-a836-cb2b1ab0fd2a","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.489761139Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"Upstairs opts: Upstairs UUID: 66b67456-fe07-431b-83a3-40c081601672, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.489807501Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"Crucible stats registered with UUID: 66b67456-fe07-431b-83a3-40c081601672","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.489834474Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:33.489864246Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.490393498Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"task":"crutest"}
{"msg":"The guest has requested activation with gen:188","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.490429166Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934}
{"msg":"66b67456-fe07-431b-83a3-40c081601672 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.490467032Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.490483762Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"0","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.49051847Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"0","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.490553769Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"1","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.490582311Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"1","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.490608605Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"2","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.490637927Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"2","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.49066514Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.490692523Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"io task","client":"1","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.490723384Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"io task","client":"0","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"ds_connection connected from Ok(127.0.0.1:43560)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.490757293Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"io task","client":"2","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"ds_connection connected from Ok(127.0.0.1:51942)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.490786755Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"io task","client":"1","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"ds_connection connected from Ok(127.0.0.1:52602)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.490817565Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"io task","client":"0","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.491630522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"0","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.491658824Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"1","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.492324972Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"1","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.492356043Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"0","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.492371203Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"2","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.492972801Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493000204Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"2","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493014984Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"[0]R flush_numbers[0..12]: [342, 326, 340, 343, 344, 344, 336, 326, 326, 326, 329, 342]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493563155Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"downstairs","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"[0]R generation[0..12]: [186, 178, 184, 186, 187, 187, 183, 178, 178, 178, 180, 186]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493590858Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"downstairs","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493606668Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"downstairs","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"[1]R flush_numbers[0..12]: [342, 326, 340, 343, 344, 344, 336, 326, 326, 326, 329, 342]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.49362052Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"downstairs","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"[1]R generation[0..12]: [186, 178, 184, 186, 187, 187, 183, 178, 178, 178, 180, 186]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493634851Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"downstairs","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493648492Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"downstairs","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"[2]R flush_numbers[0..12]: [342, 326, 340, 343, 344, 344, 336, 326, 326, 326, 329, 342]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493666681Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"downstairs","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"[2]R generation[0..12]: [186, 178, 184, 186, 187, 187, 183, 178, 178, 178, 180, 186]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493680443Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"downstairs","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493693994Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO
2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"Max found gen is 188","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493715031Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"downstairs","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"Generation requested: 188 >= found:188","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493728762Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"downstairs","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"Next flush: 345","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493741485Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"downstairs","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493754197Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"":"downstairs","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493767549Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493779221Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"0","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493792393Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"1","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493805365Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"client":"2","":"downstairs","session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493818277Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.49383011Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"66b67456-fe07-431b-83a3-40c081601672 is now active with session: 2014967c-3f69-46b9-a836-cb2b1ab0fd2a","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493842522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493854105Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934,"session_id":"2014967c-3f69-46b9-a836-cb2b1ab0fd2a"}
{"msg":"The guest has finished waiting for activation with:188","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:33.493868086Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 178 len 8192 data: 18 19
02/30 Read block 198 len 8192
03/30 Read block 180 len 12288
04/30 Write block 219 len 8192 data: 11 9
05/30 Write block 584 len 8192 data: 5 8
06/30 Read block 434 len 12288
07/30 Write block 133 len 12288 data: 10 13 12
08/30 Write block 113 len 24576 data: 13 10 14 18 22 19
09/30 Write block 456 len 4096 data: 14
10/30 Write block 215 len 32768 data: 14 11 11 12 12 10 6 7
11/30 Read block 244 len 36864
12/30 Read block 193 len 36864
13/30 Read block 473 len 40960
14/30 Read block 553 len 4096
15/30 Write block 216 len 12288 data: 12 12 13
16/30 Read block 39 len 36864
17/30 Read block 149 len 40960
18/30 Write block 542 len 28672 data: 14 15 13 12 13 11 12
19/30 Read block 408 len 8192
20/30 Read block 472 len 28672
21/30 Write block 372 len 24576 data: 17 20 21 18 19 19
22/30 Read block 109 len 4096
23/30 Read block 152 len 36864
24/30 Read block 190 len 40960
25/30 Read block 292 len 16384
26/30 Write block 395 len 12288 data: 15 14 13
27/30 Read block 35 len 36864
28/30 Write block 457 len 8192 data: 11 9
29/30 Write block 319 len 24576 data: 14 12 15 17 19 20
30/30 Write block 166 len 20480 data: 21 19 15 10 13
----------------------------------------------------------------
Crucible gen:188 GIO:true work queues: Upstairs:0 downstairs:30
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Write 2 Done Done Done false
1001 Acked Read 2 Done Done Done false
1002 Acked Read 3 Done Done Done false
1003 Acked Write 2 Done Done Done false
1004 Acked Write 2 Done Done Done false
1005 Acked Read 3 Done Done Done false
1006 Acked Write 3 Done Done Done false
1007 Acked Write 6 Done Done Done false
1008 Acked Write 1 Done Done Done false
1009 Acked Write 8 Done Done Done false
1010 Acked Read 9 Done Done Done false
1011 Acked Read 9 Done Done Done false
1012 Acked Read 10 Done Done Done false
1013 Acked Read 1 Done Done Done false
1014 Acked Write 3 Done Done Done false
1015 Acked Read 9 Done Done Done false
1016 Acked Read 10 Done Done Done false
1017 Acked Write 7 Done Done Done false
1018 Acked Read 2 Done Done Done false
1019 Acked Read 7 Done Done Done false
1020 Acked Write 6 Done Done Done false
1021 Acked Read 1 Done Done Done false
1022 Acked Read 9 Done Done Done false
1023 Acked Read 10 Done Done Done false
1024 Acked Read 4 Done Done Done false
1025 Acked Write 3 Done Done Done false
1026 Acked Read 9 Done Done Done false
1027 Acked Write 2 Done Done Done false
1028 Acked Write 6 Sent Sent Sent false
1029 Acked Write 5 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 28 28 28 84
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 0 0 0
Downstairs last five completed:
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:33.513126561Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1934}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 186 186 186 342 342 342 F F F
1 020-039 178 178 178 326 326 326 F F F
2 040-059 184 184 184 340 340 340 F F F
3 060-079 186 186 186 343 343 343 F F F
4 080-099 187 187 187 344 344 344 F F F
5 100-119 187 187 187 344 344 344 T T T
6 120-139 183 183 183 336 336 336 T T T
7 140-159 178 178 178 326 326 326 F F F
8 160-179 178 178 178 326 326 326 T T T
9 180-199 178 178 178 326 326 326 F F F
10 200-219 180 180 180 329 329 329 T T T
11 220-239 186 186 186 342 342 342 T T T
12 240-259 186 186 186 342 342 342 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 186 186 186 342 342 342 F F F
15 300-319 185 185 185 341 341 341 T T T
16 320-339 181 181 181 330 330 330 T T T
17 340-359 169 169 169 311 311 311 F F F
18 360-379 180 180 180 329 329 329 T T T
19 380-399 181 181 181 330 330 330 T T T
20 400-419 184 184 184 337 337 337 F F F
21 420-439 186 186 186 342 342 342 F F F
22 440-459 186 186 186 342 342 342 T T T
23 460-479 187 187 187 344 344 344 F F F
24 480-499 187 187 187 344 344 344 F F F
25 500-519 187 187 187 344 344 344 F F F
26 520-539 182 182 182 331 331 331 F F F
27 540-559 182 182 182 335 335 335 T T T
28 560-579 187 187 187 344 344 344 F F F
29 580-599 187 187 187 344 344 344 T T T
Max gen: 187, Max flush: 344
A Difference in extent metadata is expected here
On loop 94
Verifying data now
No change after verify
Loop: 94 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 186 186 186 342 342 342 F F F
1 020-039 178 178 178 326 326 326 F F F
2 040-059 184 184 184 340 340 340 F F F
3 060-079 186 186 186 343 343 343 F F F
4 080-099 187 187 187 344 344 344 F F F
5 100-119 188 188 188 345 345 345 F F F
6 120-139 188 188 188 345 345 345 F F F
7 140-159 178 178 178 326 326 326 F F F
8 160-179 188 188 188 345 345 345 F F F
9 180-199 178 178 178 326 326 326 F F F
10 200-219 188 188 188 345 345 345 F F F
11 220-239 188 188 188 345 345 345 F F F
12 240-259 186 186 186 342 342 342 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 186 186 186 342 342 342 F F F
15 300-319 188 188 188 345 345 345 F F F
16 320-339 188 188 188 345 345 345 F F F
17 340-359 169 169 169 311 311 311 F F F
18 360-379 188 188 188 345 345 345 F F F
19 380-399 188 188 188 345 345 345 F F F
20 400-419 184 184 184 337 337 337 F F F
21 420-439 186 186 186 342 342 342 F F F
22 440-459 188 188 188 345 345 345 F F F
23 460-479 187 187 187 344 344 344 F F F
24 480-499 187 187 187 344 344 344 F F F
25 500-519 187 187 187 344 344 344 F F F
26 520-539 182 182 182 331 331 331 F F F
27 540-559 188 188 188 345 345 345 F F F
28 560-579 187 187 187 344 344 344 F F F
29 580-599 188 188 188 345 345 345 F F F
Max gen: 188, Max flush: 345
Begin loop 95
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.814058487Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.862591533Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.863657073Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.863702685Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.863717945Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.863733036Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942}
{"msg":"Crucible ce9a6771-d4d0-48a8-8eff-f787faafff73 has session id: 673f682b-8c0c-4034-b4f9-3749e59a7524","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.864315395Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"Upstairs opts: Upstairs UUID: ce9a6771-d4d0-48a8-8eff-f787faafff73, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.864397195Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"Crucible stats registered with UUID: ce9a6771-d4d0-48a8-8eff-f787faafff73","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.864428096Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:37.864448873Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.865028154Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"task":"crutest"}
{"msg":"The guest has requested activation with gen:190","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.865063952Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942}
{"msg":"ce9a6771-d4d0-48a8-8eff-f787faafff73 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.865145371Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.8651637Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"0","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.865188375Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"0","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.865207663Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"1","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.865222264Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"1","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.86524434Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"2","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.865306761Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"2","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.865339241Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"io task","client":"2","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.865367763Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"io task","client":"0","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.86543864Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"ds_connection connected from Ok(127.0.0.1:43069)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.865481363Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"io task","client":"1","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"ds_connection connected from Ok(127.0.0.1:55998)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.865511125Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"io task","client":"2","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49877)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.865538518Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"io task","client":"0","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.866253485Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"0","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.866301356Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"1","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.866502871Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"2","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.866980545Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"0","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867012915Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"1","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867035421Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"2","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867566742Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867664681Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"[0]R flush_numbers[0..12]: [342, 326, 340, 343, 344, 345, 345, 326, 345, 326, 345, 345]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867749779Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"downstairs","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"[0]R generation[0..12]: [186, 178, 184, 186, 187, 188, 188, 178, 188, 178, 188, 188]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867769136Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"downstairs","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867783907Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"downstairs","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"[1]R flush_numbers[0..12]: [342, 326, 340, 343, 344, 345, 345, 326, 345, 326, 345, 345]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867797509Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"downstairs","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"[1]R generation[0..12]: [186, 178, 184, 186, 187, 188, 188, 178, 188, 178, 188, 188]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867810971Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"downstairs","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867824632Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"downstairs","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"[2]R flush_numbers[0..12]: [342, 326, 340, 343, 344, 345, 345, 326, 345, 326, 345, 345]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867837914Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"downstairs","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"[2]R generation[0..12]: [186, 178, 184, 186, 187, 188, 188, 178, 188, 178, 188, 188]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867851376Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"downstairs","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867865047Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"downstairs","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"Max found gen is 189","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.86787732Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"downstairs","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"Generation requested: 190 >= found:189","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.86789318Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"downstairs","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
Wait for a query_work_queue command to finish before sending IO
{"msg":"Next flush: 346","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867912098Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"downstairs","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.86792483Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"":"downstairs","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867937502Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867949145Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"0","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867962187Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"1","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867975039Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"client":"2","":"downstairs","session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867987881Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.867999934Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"ce9a6771-d4d0-48a8-8eff-f787faafff73 is now active with session: 673f682b-8c0c-4034-b4f9-3749e59a7524","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.868012476Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.868024198Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942,"session_id":"673f682b-8c0c-4034-b4f9-3749e59a7524"}
{"msg":"The guest has finished waiting for activation with:190","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:37.868036081Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 491 len 28672 data: 19 22 19 15 20 20 16
02/30 Write block 223 len 4096 data: 10
03/30 Read block 325 len 24576
04/30 Read block 493 len 28672
05/30 Write block 295 len 36864 data: 15 17 15 12 12 12 14 13 15
06/30 Write block 215 len 40960 data: 15 13 13 14 13 11 7 8 11 12
07/30 Flush
08/30 Write block 366 len 28672 data: 13 13 17 16 17 17 18
09/30 Write block 43 len 24576 data: 11 11 11 10 8 8
10/30 Read block 508 len 4096
11/30 Write block 334 len 28672 data: 8 9 11 12 15 15 14
12/30 Write block 572 len 12288 data: 14 15 14
13/30 Read block 375 len 8192
14/30 Write block 115 len 28672 data: 15 19 23 20 21 18 19
15/30 Read block 167 len 36864
16/30 Write block 566 len 24576 data: 9 11 10 11 13 13
17/30 Write block 197 len 36864 data: 16 18 15 13 13 13 13 15 15
18/30 Write block 295 len 40960 data: 16 18 16 13 13 13 15 14 16 14
19/30 Write block 10 len 32768 data: 12 11 7 6 6 7 7 10
20/30 Read block 518 len 20480
21/30 Write block 418 len 20480 data: 16 17 18 16 18
22/30 Read block 80 len 40960
23/30 Write block 102 len 20480 data: 20 21 20 20 18
24/30 Write block 319 len 12288 data: 15 13 16
25/30 Read block 341 len 4096
26/30 Read block 532 len 16384
27/30 Write block 29 len 28672 data: 16 15 14 12 11 10 9
28/30 Read block 144 len 36864
29/30 Read block 452 len 28672
30/30 Write block 570 len 32768 data: 14 14 15 16 15 17 14 14
----------------------------------------------------------------
Crucible gen:190 GIO:true work queues: Upstairs:0 downstairs:23
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1007 Acked Write 7 Done Done Done false
1008 Acked Write 6 Done Done Done false
1009 Acked Read 1 Done Done Done false
1010 Acked Write 7 Done Done Done false
1011 Acked Write 3 Done Done Done false
1012 Acked Read 2 Done Done Done false
1013 Acked Write 7 Done Done Done false
1014 Acked Read 9 Done Done Done false
1015 Acked Write 6 Done Done Done false
1016 Acked Write 9 Done Done Done false
1017 Acked Write 10 Done Done Done false
1018 Acked Write 8 Done Done Done false
1019 Acked Read 5 Done Done Done false
1020 Acked Write 5 Done Done Done false
1021 Acked Read 10 Done Done Done false
1022 Acked Write 5 Done Done Done false
1023 Acked Write 3 Done Done Done false
1024 Acked Read 1 Done Done Done false
1025 Acked Read 4 Done Done Done false
1026 Acked Write 7 Done Done Done false
1027 Acked Read 9 Done Done Done false
1028 Acked Read 7 Done Done Done false
1029 Acked Write 8 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 1 1 1 3
Done 22 22 22 66
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1006 1006 1006
Downstairs last five completed: 1006 1005 1004 1003 1002
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:37.889156942Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1942}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 186 186 186 342 342 342 T T T
1 020-039 178 178 178 326 326 326 T T T
2 040-059 184 184 184 340 340 340 T T T
3 060-079 186 186 186 343 343 343 F F F
4 080-099 187 187 187 344 344 344 F F F
5 100-119 188 188 188 345 345 345 T T T
6 120-139 188 188 188 345 345 345 T T T
7 140-159 178 178 178 326 326 326 F F F
8 160-179 188 188 188 345 345 345 F F F
9 180-199 178 178 178 326 326 326 T T T
10 200-219 190 190 190 346 346 346 T T T
11 220-239 190 190 190 346 346 346 F F F
12 240-259 186 186 186 342 342 342 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 190 190 190 346 346 346 T T T
15 300-319 190 190 190 346 346 346 T T T
16 320-339 188 188 188 345 345 345 T T T
17 340-359 169 169 169 311 311 311 T T T
18 360-379 188 188 188 345 345 345 T T T
19 380-399 188 188 188 345 345 345 F F F
20 400-419 184 184 184 337 337 337 T T T
21 420-439 186 186 186 342 342 342 T T T
22 440-459 188 188 188 345 345 345 F F F
23 460-479 187 187 187 344 344 344 F F F
24 480-499 190 190 190 346 346 346 F F F
25 500-519 187 187 187 344 344 344 F F F
26 520-539 182 182 182 331 331 331 F F F
27 540-559 188 188 188 345 345 345 F F F
28 560-579 187 187 187 344 344 344 T T T
29 580-599 188 188 188 345 345 345 F F F
Max gen: 190, Max flush: 346
A Difference in extent metadata is expected here
On loop 95
Verifying data now
No change after verify
Loop: 95 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 191 191 191 347 347 347 F F F
1 020-039 191 191 191 347 347 347 F F F
2 040-059 191 191 191 347 347 347 F F F
3 060-079 186 186 186 343 343 343 F F F
4 080-099 187 187 187 344 344 344 F F F
5 100-119 191 191 191 347 347 347 F F F
6 120-139 191 191 191 347 347 347 F F F
7 140-159 178 178 178 326 326 326 F F F
8 160-179 188 188 188 345 345 345 F F F
9 180-199 191 191 191 347 347 347 F F F
10 200-219 191 191 191 347 347 347 F F F
11 220-239 190 190 190 346 346 346 F F F
12 240-259 186 186 186 342 342 342 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 191 191 191 347 347 347 F F F
15 300-319 191 191 191 347 347 347 F F F
16 320-339 191 191 191 347 347 347 F F F
17 340-359 191 191 191 347 347 347 F F F
18 360-379 191 191 191 347 347 347 F F F
19 380-399 188 188 188 345 345 345 F F F
20 400-419 191 191 191 347 347 347 F F F
21 420-439 191 191 191 347 347 347 F F F
22 440-459 188 188 188 345 345 345 F F F
23 460-479 187 187 187 344 344 344 F F F
24 480-499 190 190 190 346 346 346 F F F
25 500-519 187 187 187 344 344 344 F F F
26 520-539 182 182 182 331 331 331 F F F
27 540-559 188 188 188 345 345 345 F F F
28 560-579 191 191 191 347 347 347 F F F
29 580-599 188 188 188 345 345 345 F F F
Max gen: 191, Max flush: 347
Begin loop 96
Downstairs to restart: 2
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.209201434Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.255787926Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.256819587Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.256873843Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.256891602Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.256909331Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950}
{"msg":"Crucible dd645626-5078-4624-9ef4-5f07391bae1f has session id: 66189f61-f1cc-45ef-a9b8-8c76a5cafe8b","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.257533914Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"Upstairs opts: Upstairs UUID: dd645626-5078-4624-9ef4-5f07391bae1f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.257578717Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"Crucible stats registered with UUID: dd645626-5078-4624-9ef4-5f07391bae1f","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.257601543Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:42.257621Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258159027Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"task":"crutest"}
{"msg":"The guest has requested activation with gen:192","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258185711Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950}
{"msg":"dd645626-5078-4624-9ef4-5f07391bae1f active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258248602Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.25826704Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"0","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258292635Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"0","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258306186Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"1","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258320347Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"1","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258333339Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"2","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258347071Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"2","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258361102Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"io task","client":"2","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258378602Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258408683Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"io task","client":"1","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"ds_connection connected from Ok(127.0.0.1:65484)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258791546Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"io task","client":"1","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47391)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258845182Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"io task","client":"2","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"ds_connection connected from Ok(127.0.0.1:44401)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.258878682Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"io task","client":"0","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.259530168Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"1","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.259578069Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"0","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.259679456Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"2","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260144208Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"1","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260178097Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"0","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260244166Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"2","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260715594Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260741918Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"[0]R flush_numbers[0..12]: [347, 347, 347, 343, 344, 347, 347, 326, 345, 347, 347, 346]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260769661Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"downstairs","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"[0]R generation[0..12]: [191, 191, 191, 186, 187, 191, 191, 178, 188, 191, 191, 190]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260784512Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"downstairs","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260802421Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"downstairs","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"[1]R flush_numbers[0..12]: [347, 347, 347, 343, 344, 347, 347, 326, 345, 347, 347, 346]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260816832Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"downstairs","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"[1]R generation[0..12]: [191, 191, 191, 186, 187, 191, 191, 178, 188, 191, 191, 190]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260830393Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"downstairs","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260844105Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"downstairs","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"[2]R flush_numbers[0..12]: [347, 347, 347, 343, 344, 347, 347, 326, 345, 347, 347, 346]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260857966Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"downstairs","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"[2]R generation[0..12]: [191, 191, 191, 186, 187, 191, 191, 178, 188, 191, 191, 190]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260871428Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"downstairs","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260885099Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"downstairs","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"Max found gen is 192","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260897851Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"downstairs","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"Generation requested: 192 >= found:192","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260910743Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"downstairs","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"Next flush: 348","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260923296Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"downstairs","":"Wait for a query_work_queue command to finish before sending IO
downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260942943Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"":"downstairs","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260955646Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260967259Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"0","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.26098008Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"1","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.260992703Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"client":"2","":"downstairs","session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.261007463Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.261019676Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"dd645626-5078-4624-9ef4-5f07391bae1f is now active with session: 66189f61-f1cc-45ef-a9b8-8c76a5cafe8b","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.261032048Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.261043761Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950,"session_id":"66189f61-f1cc-45ef-a9b8-8c76a5cafe8b"}
{"msg":"The guest has finished waiting for activation with:192","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:42.261055714Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 72 len 40960 data: 22 24 23 22 19 13 14 11 11 12
02/30 Read block 22 len 4096
03/30 Read block 552 len 8192
04/30 Write block 68 len 32768 data: 19 17 23 27 23 25 24 23
05/30 Read block 117 len 16384
06/30 Write block 393 len 24576 data: 17 16 16 15 14 11
07/30 Read block 194 len 12288
08/30 Write block 30 len 12288 data: 16 15 13
09/30 Write block 38 len 24576 data: 9 11 13 15 12 12
10/30 Read block 312 len 28672
11/30 Write block 379 len 40960 data: 17 12 11 10 10 10 10 13 12 15
12/30 Write block 499 len 24576 data: 15 10 9 11 10 11
13/30 Write block 235 len 28672 data: 10 8 7 6 6 9 12
14/30 Read block 288 len 32768
15/30 Read block 20 len 40960
16/30 Read block 574 len 28672
17/30 Write block 586 len 4096 data: 10
18/30 Write block 238 len 24576 data: 7 7 10 13 14 16
19/30 Read block 422 len 8192
20/30 Flush
21/30 Write block 22 len 24576 data: 15 17 19 17 17 17
22/30 Write block 152 len 16384 data: 13 19 18 14
23/30 Read block 466 len 12288
24/30 Read block 59 len 28672
25/30 Read block 101 len 32768
26/30 Read block 223 len 32768
27/30 Read block 374 len 12288
28/30 Write block 103 len 36864 data: 22 21 21 19 14 13 13 12 11
29/30 Read block 152 len 24576
30/30 Read block 68 len 36864
----------------------------------------------------------------
Crucible gen:192 GIO:true work queues: Upstairs:0 downstairs:10
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1020 Acked Write 6 Done Done Done false
1021 Acked Write 4 Done Done Done false
1022 Acked Read 3 Done Done Done false
1023 Acked Read 7 Done Done Done false
1024 Acked Read 8 Done Done Done false
1025 Acked Read 8 Done Done Done false
1026 Acked Read 3 Done Done Done false
1027 Acked Write 9 Done Done Done false
1028 Acked Read 6 Done Done Done false
1029 Acked Read 9 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 10 10 10 30
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1019 1019 1019
Downstairs last five completed: 1019 1018 1017 1016 1015
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:42.282602461Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1950}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 191 191 191 347 347 347 F F F
1 020-039 192 192 192 348 348 348 T T T
2 040-059 192 192 192 348 348 348 F F F
3 060-079 192 192 192 348 348 348 F F F
4 080-099 192 192 192 348 348 348 F F F
5 100-119 191 191 191 347 347 347 T T T
6 120-139 191 191 191 347 347 347 F F F
7 140-159 178 178 178 326 326 326 T T T
8 160-179 188 188 188 345 345 345 F F F
9 180-199 191 191 191 347 347 347 F F F
10 200-219 191 191 191 347 347 347 F F F
11 220-239 192 192 192 348 348 348 F F F
12 240-259 192 192 192 348 348 348 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 191 191 191 347 347 347 F F F
15 300-319 191 191 191 347 347 347 F F F
16 320-339 191 191 191 347 347 347 F F F
17 340-359 191 191 191 347 347 347 F F F
18 360-379 192 192 192 348 348 348 F F F
19 380-399 192 192 192 348 348 348 F F F
20 400-419 191 191 191 347 347 347 F F F
21 420-439 191 191 191 347 347 347 F F F
22 440-459 188 188 188 345 345 345 F F F
23 460-479 187 187 187 344 344 344 F F F
24 480-499 192 192 192 348 348 348 F F F
25 500-519 192 192 192 348 348 348 F F F
26 520-539 182 182 182 331 331 331 F F F
27 540-559 188 188 188 345 345 345 F F F
28 560-579 191 191 191 347 347 347 F F F
29 580-599 192 192 192 348 348 348 F F F
Max gen: 192, Max flush: 348
A Difference in extent metadata is expected here
On loop 96
Verifying data now
No change after verify
Loop: 96 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 191 191 191 347 347 347 F F F
1 020-039 193 193 193 349 349 349 F F F
2 040-059 192 192 192 348 348 348 F F F
3 060-079 192 192 192 348 348 348 F F F
4 080-099 192 192 192 348 348 348 F F F
5 100-119 193 193 193 349 349 349 F F F
6 120-139 191 191 191 347 347 347 F F F
7 140-159 193 193 193 349 349 349 F F F
8 160-179 188 188 188 345 345 345 F F F
9 180-199 191 191 191 347 347 347 F F F
10 200-219 191 191 191 347 347 347 F F F
11 220-239 192 192 192 348 348 348 F F F
12 240-259 192 192 192 348 348 348 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 191 191 191 347 347 347 F F F
15 300-319 191 191 191 347 347 347 F F F
16 320-339 191 191 191 347 347 347 F F F
17 340-359 191 191 191 347 347 347 F F F
18 360-379 192 192 192 348 348 348 F F F
19 380-399 192 192 192 348 348 348 F F F
20 400-419 191 191 191 347 347 347 F F F
21 420-439 191 191 191 347 347 347 F F F
22 440-459 188 188 188 345 345 345 F F F
23 460-479 187 187 187 344 344 344 F F F
24 480-499 192 192 192 348 348 348 F F F
25 500-519 192 192 192 348 348 348 F F F
26 520-539 182 182 182 331 331 331 F F F
27 540-559 188 188 188 345 345 345 F F F
28 560-579 191 191 191 347 347 347 F F F
29 580-599 192 192 192 348 348 348 F F F
Max gen: 193, Max flush: 349
Begin loop 97
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.524304145Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.569712686Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.570746526Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.570790259Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.570807118Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.570821679Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958}
{"msg":"Crucible 19b76041-82d5-4e4a-948c-d24be5fa9724 has session id: 3b86bcd1-f793-4a26-b51f-8f38deac6a48","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.571552507Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"Upstairs opts: Upstairs UUID: 19b76041-82d5-4e4a-948c-d24be5fa9724, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.571589294Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"Crucible stats registered with UUID: 19b76041-82d5-4e4a-948c-d24be5fa9724","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.571604384Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:46.571617187Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.572140083Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"task":"crutest"}
{"msg":"The guest has requested activation with gen:194","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.572170134Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958}
{"msg":"19b76041-82d5-4e4a-948c-d24be5fa9724 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.572231166Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.57225748Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"0","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.572284783Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"0","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.572315754Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"1","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.572345995Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"1","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.572373368Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"2","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.572399632Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"2","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.572424576Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.5724506Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"io task","client":"0","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.572477144Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"io task","client":"1","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47377)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.572502798Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"io task","client":"2","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"ds_connection connected from Ok(127.0.0.1:33305)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.572526633Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"io task","client":"0","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"ds_connection connected from Ok(127.0.0.1:59927)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.572549879Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"io task","client":"1","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.573240641Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"2","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.573269044Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"0","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.573863855Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"0","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.573892028Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"2","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.574408938Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"1","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.574570568Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.574600259Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.574942647Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"1","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"[0]R flush_numbers[0..12]: [347, 349, 348, 348, 348, 349, 347, 349, 345, 347, 347, 348]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.57545418Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"downstairs","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"[0]R generation[0..12]: [191, 193, 192, 192, 192, 193, 191, 193, 188, 191, 191, 192]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575480604Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"downstairs","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575495915Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"downstairs","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"[1]R flush_numbers[0..12]: [347, 349, 348, 348, 348, 349, 347, 349, 345, 347, 347, 348]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575509876Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"downstairs","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"[1]R generation[0..12]: [191, 193, 192, 192, 192, 193, 191, 193, 188, 191, 191, 192]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575523438Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"downstairs","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575537549Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"downstairs","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"[2]R flush_numbers[0..12]: [347, 349, 348, 348, 348, 349, 347, 349, 345, 347, 347, 348]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.57555124Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"downstairs","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"[2]R generation[0..12]: [191, 193, 192, 192, 192, 193, 191, 193, 188, 191, 191, 192]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575564872Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"downstairs","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575578394Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"downstairs","":"downstairs","session_id":"Wait for a query_work_queue command to finish before sending IO
3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"Max found gen is 194","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575598161Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"downstairs","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"Generation requested: 194 >= found:194","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575614401Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"downstairs","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"Next flush: 350","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575627133Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"downstairs","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575639516Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"":"downstairs","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575652158Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575663761Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"0","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575676593Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"1","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575689225Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"client":"2","":"downstairs","session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575701937Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.57571376Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"19b76041-82d5-4e4a-948c-d24be5fa9724 is now active with session: 3b86bcd1-f793-4a26-b51f-8f38deac6a48","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575726182Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575737765Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958,"session_id":"3b86bcd1-f793-4a26-b51f-8f38deac6a48"}
{"msg":"The guest has finished waiting for activation with:194","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:46.575749568Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 537 len 32768 data: 17 15 16 15 15 15 16 14
02/30 Write block 82 len 16384 data: 15 14 18 22
03/30 Write block 468 len 8192 data: 12 10
04/30 Flush
05/30 Write block 361 len 36864 data: 15 16 13 11 10 14 14 18 17
06/30 Write block 92 len 36864 data: 22 26 21 20 19 20 21 20 20
07/30 Write block 46 len 12288 data: 11 9 9
08/30 Write block 559 len 32768 data: 15 14 10 8 9 10 10 10
09/30 Read block 481 len 8192
10/30 Write block 68 len 16384 data: 20 18 24 28
11/30 Read block 344 len 28672
12/30 Write block 301 len 4096 data: 16
13/30 Write block 503 len 24576 data: 11 12 12 14 14 14
14/30 Read block 13 len 8192
15/30 Read block 385 len 40960
16/30 Read block 177 len 16384
17/30 Read block 21 len 24576
18/30 Read block 444 len 36864
19/30 Read block 490 len 40960
20/30 Read block 355 len 32768
21/30 Write block 216 len 4096 data: 14
22/30 Flush
23/30 Write block 166 len 36864 data: 22 20 16 11 14 12 11 10 11
24/30 Write block 419 len 40960 data: 18 19 17 19 19 15 14 12 12 9
25/30 Write block 419 len 4096 data: 19
26/30 Read block 22 len 32768
27/30 Read block 50 len 32768
28/30 Read block 269 len 4096
29/30 Write block 462 len 4096 data: 17
30/30 Read block 132 len 20480
----------------------------------------------------------------
Crucible gen:194 GIO:true work queues: Upstairs:0 downstairs:8
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1022 Acked Write 9 Done Done Done false
1023 Acked Write 10 Done Done Done false
1024 Acked Write 1 Done Done Done false
1025 Acked Read 8 Done Done Done false
1026 Acked Read 8 Done Done Done false
1027 Acked Read 1 Done Done Done false
1028 Acked Write 1 Done Sent Done false
1029 Acked Read 5 Done Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 2 1 3
Done 8 6 7 21
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1021 1021 1021
Downstairs last five completed: 1021 1020 1019 1018 1017
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:46.595157719Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1958}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 191 191 191 347 347 347 F F F
1 020-039 193 193 193 349 349 349 F F F
2 040-059 194 194 194 351 351 351 F F F
3 060-079 194 194 194 351 351 351 F F F
4 080-099 194 194 194 351 351 351 F F F
5 100-119 194 194 194 351 351 351 F F F
6 120-139 191 191 191 347 347 347 F F F
7 140-159 193 193 193 349 349 349 F F F
8 160-179 188 188 188 345 345 345 T T T
9 180-199 191 191 191 347 347 347 F F F
10 200-219 194 194 194 351 351 351 F F F
11 220-239 192 192 192 348 348 348 F F F
12 240-259 192 192 192 348 348 348 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 191 191 191 347 347 347 F F F
15 300-319 194 194 194 351 351 351 F F F
16 320-339 191 191 191 347 347 347 F F F
17 340-359 191 191 191 347 347 347 F F F
18 360-379 194 194 194 351 351 351 F F F
19 380-399 192 192 192 348 348 348 F F F
20 400-419 191 191 191 347 347 347 T T T
21 420-439 191 191 191 347 347 347 T T T
22 440-459 188 188 188 345 345 345 F F F
23 460-479 194 194 194 350 350 350 T T T
24 480-499 192 192 192 348 348 348 F F F
25 500-519 194 194 194 351 351 351 F F F
26 520-539 194 194 194 350 350 350 F F F
27 540-559 194 194 194 351 351 351 F F F
28 560-579 194 194 194 351 351 351 F F F
29 580-599 192 192 192 348 348 348 F F F
Max gen: 194, Max flush: 351
A Difference in extent metadata is expected here
On loop 97
Verifying data now
No change after verify
Loop: 97 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 191 191 191 347 347 347 F F F
1 020-039 193 193 193 349 349 349 F F F
2 040-059 194 194 194 351 351 351 F F F
3 060-079 194 194 194 351 351 351 F F F
4 080-099 194 194 194 351 351 351 F F F
5 100-119 194 194 194 351 351 351 F F F
6 120-139 191 191 191 347 347 347 F F F
7 140-159 193 193 193 349 349 349 F F F
8 160-179 195 195 195 352 352 352 F F F
9 180-199 191 191 191 347 347 347 F F F
10 200-219 194 194 194 351 351 351 F F F
11 220-239 192 192 192 348 348 348 F F F
12 240-259 192 192 192 348 348 348 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 191 191 191 347 347 347 F F F
15 300-319 194 194 194 351 351 351 F F F
16 320-339 191 191 191 347 347 347 F F F
17 340-359 191 191 191 347 347 347 F F F
18 360-379 194 194 194 351 351 351 F F F
19 380-399 192 192 192 348 348 348 F F F
20 400-419 195 195 195 352 352 352 F F F
21 420-439 195 195 195 352 352 352 F F F
22 440-459 188 188 188 345 345 345 F F F
23 460-479 195 195 195 352 352 352 F F F
24 480-499 192 192 192 348 348 348 F F F
25 500-519 194 194 194 351 351 351 F F F
26 520-539 194 194 194 350 350 350 F F F
27 540-559 194 194 194 351 351 351 F F F
28 560-579 194 194 194 351 351 351 F F F
29 580-599 192 192 192 348 348 348 F F F
Max gen: 195, Max flush: 352
Begin loop 98
Downstairs to restart: 0
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.837785449Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:50.881452888Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.882123843Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.914605079Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.915697002Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.915742374Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.915759414Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.915776943Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966}
{"msg":"Crucible 9d1b9f25-3e34-4e05-8122-c7b3da53c27d has session id: c2c5abad-95a5-49f0-96a9-7e4b087c20a9","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.9165404Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"Upstairs opts: Upstairs UUID: 9d1b9f25-3e34-4e05-8122-c7b3da53c27d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.916575958Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"Crucible stats registered with UUID: 9d1b9f25-3e34-4e05-8122-c7b3da53c27d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.916597235Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:50.916610607Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.917142347Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"task":"crutest"}
{"msg":"The guest has requested activation with gen:196","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.917175767Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966}
{"msg":"9d1b9f25-3e34-4e05-8122-c7b3da53c27d active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.91723491Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.91726649Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"0","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.917289956Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"0","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.917304847Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"1","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.917319957Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"1","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.917333349Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"2","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.91734777Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"2","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.917365479Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"io task","client":"2","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.917387506Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"io task","client":"0","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.917410241Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"ds_connection connected from Ok(127.0.0.1:49440)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.917596526Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"io task","client":"2","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"ds_connection connected from Ok(127.0.0.1:35655)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.917650543Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"io task","client":"1","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"ds_connection connected from Ok(127.0.0.1:63019)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.917691797Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"io task","client":"0","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.918382769Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"1","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.91841437Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"2","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.918886557Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"0","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.919125179Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"1","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.91935014Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"2","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.919519186Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"0","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.919781973Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.919848162Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"[0]R flush_numbers[0..12]: [347, 349, 351, 351, 351, 351, 347, 349, 352, 347, 351, 348]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920150005Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"downstairs","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"[0]R generation[0..12]: [191, 193, 194, 194, 194, 194, 191, 193, 195, 191, 194, 192]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920172521Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"downstairs","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920187522Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"downstairs","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"[1]R flush_numbers[0..12]: [347, 349, 351, 351, 351, 351, 347, 349, 352, 347, 351, 348]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920201973Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"downstairs","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"[1]R generation[0..12]: [191, 193, 194, 194, 194, 194, 191, 193, 195, 191, 194, 192]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920218922Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"downstairs","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920232794Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"downstairs","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"[2]R flush_numbers[0..12]: [347, 349, 351, 351, 351, 351, 347, 349, 352, 347, 351, 348]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920246555Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"downstairs","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"[2]R generation[0..12]: [191, 193, 194, 194, 194, 194, 191, 193, 195, 191, 194, 192]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920260227Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"downstairs","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920273948Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"downstairs","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"Max found gen is 196","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.9202866Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"downstairs","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"Generation requested: 196 >= found:196","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920299562Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"downstairs","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"Next flush: 353","v":0,"name":"Wait for a query_work_queue command to finish before sending IO
crucible","level":30,"time":"2025-01-27T19:44:50.920319Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"downstairs","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920332482Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"":"downstairs","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920345884Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920358836Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"0","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920372457Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"1","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920385769Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"client":"2","":"downstairs","session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920399211Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920411623Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"9d1b9f25-3e34-4e05-8122-c7b3da53c27d is now active with session: c2c5abad-95a5-49f0-96a9-7e4b087c20a9","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920426584Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920438926Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966,"session_id":"c2c5abad-95a5-49f0-96a9-7e4b087c20a9"}
{"msg":"The guest has finished waiting for activation with:196","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:50.920451258Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 435 len 36864 data: 21 17 12 10 10 11 11 11 11
02/30 Write block 585 len 28672 data: 9 11 12 13 14 15 13
03/30 Flush
04/30 Flush
05/30 Write block 165 len 32768 data: 19 23 21 17 12 15 13 12
06/30 Flush
07/30 Write block 538 len 8192 data: 16 17
08/30 Write block 220 len 20480 data: 12 8 9 12 13
09/30 Write block 531 len 20480 data: 16 15 17 17 20
10/30 Write block 364 len 16384 data: 12 11 15 15
11/30 Read block 206 len 12288
12/30 Read block 179 len 12288
13/30 Read block 434 len 36864
14/30 Write block 250 len 40960 data: 13 11 11 13 13 13 12 13 11 11
15/30 Read block 585 len 40960
16/30 Write block 575 len 40960 data: 18 15 15 12 11 13 12 9 8 6
17/30 Read block 543 len 40960
18/30 Flush
19/30 Write block 148 len 4096 data: 16
20/30 Write block 580 len 24576 data: 14 13 10 9 7 10
21/30 Write block 60 len 8192 data: 15 15
22/30 Write block 97 len 16384 data: 21 22 21 21
23/30 Write block 519 len 36864 data: 12 13 15 18 16 16 15 13 14
24/30 Read block 412 len 20480
25/30 Write block 74 len 36864 data: 25 24 20 14 15 12 12 13 16
26/30 Read block 8 len 20480
27/30 Write block 531 len 28672 data: 17 16 18 18 21 16 18
28/30 Read block 524 len 32768
29/30 Write block 597 len 12288 data: 9 6 3
30/30 Write block 212 len 4096 data: 15
----------------------------------------------------------------
Crucible gen:196 GIO:true work queues: Upstairs:0 downstairs:12
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1018 Acked Write 1 Done Done Done false
1019 Acked Write 6 Done Done Done false
1020 Acked Write 2 Done Done Done false
1021 Acked Write 4 Done Done Done false
1022 Acked Write 9 Done Done Done false
1023 Acked Read 5 Done Done Done false
1024 Acked Write 9 Done Done Done false
1025 Acked Read 5 Done Done Done false
1026 Acked Write 7 Done Done Done false
1027 Acked Read 8 Done Done Done false
1028 Acked Write 3 Sent Sent Sent false
1029 Acked Write 1 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 10 10 10 30
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1017 1017 1017
Downstairs last five completed: 1017 1016 1015 1014 1013
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:50.940339713Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1966}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 191 191 191 347 347 347 F F F
1 020-039 193 193 193 349 349 349 F F F
2 040-059 194 194 194 351 351 351 F F F
3 060-079 194 194 194 351 351 351 T T T
4 080-099 194 194 194 351 351 351 T T T
5 100-119 194 194 194 351 351 351 T T T
6 120-139 191 191 191 347 347 347 F F F
7 140-159 193 193 193 349 349 349 T T T
8 160-179 196 196 196 355 355 355 F F F
9 180-199 191 191 191 347 347 347 F F F
10 200-219 194 194 194 351 351 351 T T T
11 220-239 196 196 196 356 356 356 F F F
12 240-259 196 196 196 356 356 356 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 191 191 191 347 347 347 F F F
15 300-319 194 194 194 351 351 351 F F F
16 320-339 191 191 191 347 347 347 F F F
17 340-359 191 191 191 347 347 347 F F F
18 360-379 196 196 196 356 356 356 F F F
19 380-399 192 192 192 348 348 348 F F F
20 400-419 195 195 195 352 352 352 F F F
21 420-439 196 196 196 353 353 353 F F F
22 440-459 196 196 196 353 353 353 F F F
23 460-479 195 195 195 352 352 352 F F F
24 480-499 192 192 192 348 348 348 F F F
25 500-519 194 194 194 351 351 351 T T T
26 520-539 196 196 196 356 356 356 T T T
27 540-559 194 194 194 351 351 351 F F F
28 560-579 196 196 196 356 356 356 F F F
29 580-599 196 196 196 356 356 356 T T T
Max gen: 196, Max flush: 356
A Difference in extent metadata is expected here
On loop 98
Verifying data now
No change after verify
Loop: 98 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 191 191 191 347 347 347 F F F
1 020-039 193 193 193 349 349 349 F F F
2 040-059 194 194 194 351 351 351 F F F
3 060-079 197 197 197 357 357 357 F F F
4 080-099 197 197 197 357 357 357 F F F
5 100-119 197 197 197 357 357 357 F F F
6 120-139 191 191 191 347 347 347 F F F
7 140-159 197 197 197 357 357 357 F F F
8 160-179 196 196 196 355 355 355 F F F
9 180-199 191 191 191 347 347 347 F F F
10 200-219 197 197 197 357 357 357 F F F
11 220-239 196 196 196 356 356 356 F F F
12 240-259 196 196 196 356 356 356 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 191 191 191 347 347 347 F F F
15 300-319 194 194 194 351 351 351 F F F
16 320-339 191 191 191 347 347 347 F F F
17 340-359 191 191 191 347 347 347 F F F
18 360-379 196 196 196 356 356 356 F F F
19 380-399 192 192 192 348 348 348 F F F
20 400-419 195 195 195 352 352 352 F F F
21 420-439 196 196 196 353 353 353 F F F
22 440-459 196 196 196 353 353 353 F F F
23 460-479 195 195 195 352 352 352 F F F
24 480-499 192 192 192 348 348 348 F F F
25 500-519 197 197 197 357 357 357 F F F
26 520-539 197 197 197 357 357 357 F F F
27 540-559 194 194 194 351 351 351 F F F
28 560-579 196 196 196 356 356 356 F F F
29 580-599 197 197 197 357 357 357 F F F
Max gen: 197, Max flush: 357
Begin loop 99
Downstairs to restart: 1
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.23599091Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.282689821Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.283598809Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"d77aa90b6afe7d7ddb1dc63211fa4216c8866a90\",\n git_commit_timestamp: \"2025-01-27T19:20:16.000000000Z\",\n git_branch: \"mkeeter/shrink-replay-buffer\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.283642822Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974}
{"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.283659671Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.28367707Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974}
{"msg":"Crucible c6e6c33f-fee8-4e09-8053-3e3d2e868c4f has session id: 8cbdb453-f72b-46cd-b2f2-4b41ecce841d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.28437092Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"Upstairs opts: Upstairs UUID: c6e6c33f-fee8-4e09-8053-3e3d2e868c4f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.28442085Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"Crucible stats registered with UUID: c6e6c33f-fee8-4e09-8053-3e3d2e868c4f","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.284440597Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:55.284456478Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.284865674Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"task":"crutest"}
{"msg":"The guest has requested activation with gen:198","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.28488935Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974}
{"msg":"c6e6c33f-fee8-4e09-8053-3e3d2e868c4f active request set","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.284972878Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.284989517Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"0","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.285012663Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"0","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.285025925Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"1","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.285042085Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"1","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.285054867Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"2","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.285069538Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"2","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.28508233Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.285095802Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"io task","client":"0","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.285111922Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"io task","client":"1","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:36572)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.285321182Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"io task","client":"1","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:44692)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.285392108Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"io task","client":"2","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"ds_connection connected from Ok(127.0.0.1:60713)","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.28542205Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"io task","client":"0","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.286047872Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"0","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.286076045Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"2","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.28634267Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"1","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.286819164Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"2","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.286847367Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"0","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.286861988Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"1","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287448925Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287474299Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"[0]R flush_numbers[0..12]: [347, 349, 351, 357, 357, 357, 347, 357, 355, 347, 357, 356]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287540268Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"downstairs","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"[0]R generation[0..12]: [191, 193, 194, 197, 197, 197, 191, 197, 196, 191, 197, 196]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287559896Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"downstairs","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287573907Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"downstairs","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"[1]R flush_numbers[0..12]: [347, 349, 351, 357, 357, 357, 347, 357, 355, 347, 357, 356]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287587459Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"downstairs","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"[1]R generation[0..12]: [191, 193, 194, 197, 197, 197, 191, 197, 196, 191, 197, 196]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.28760096Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"downstairs","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287614872Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"downstairs","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"[2]R flush_numbers[0..12]: [347, 349, 351, 357, 357, 357, 347, 357, 355, 347, 357, 356]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287628363Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"downstairs","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"[2]R generation[0..12]: [191, 193, 194, 197, 197, 197, 191, 197, 196, 191, 197, 196]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287641405Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"downstairs","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287654657Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"downstairs","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"Max found gen is 198","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287667019Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"downstairs","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"Generation requested: 198 >= found:198","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287679862Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"downstairs","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"Next flush: 358","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287693073Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"downstairs","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
Wait for a query_work_queue command to finish before sending IO
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.28771506Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"":"downstairs","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287728332Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287740164Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"0","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287753156Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"1","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287765858Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"client":"2","":"downstairs","session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.28777887Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287790873Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"c6e6c33f-fee8-4e09-8053-3e3d2e868c4f is now active with session: 8cbdb453-f72b-46cd-b2f2-4b41ecce841d","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287803085Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287814578Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974,"session_id":"8cbdb453-f72b-46cd-b2f2-4b41ecce841d"}
{"msg":"The guest has finished waiting for activation with:198","v":0,"name":"crucible","level":30,"time":"2025-01-27T19:44:55.287826411Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Run Repair workload
01/30 Write block 196 len 32768 data: 14 17 19 16 14 14 14 14
02/30 Read block 403 len 36864
03/30 Read block 94 len 40960
04/30 Read block 581 len 24576
05/30 Flush
06/30 Write block 461 len 36864 data: 12 18 15 15 14 13 13 13 11
07/30 Write block 146 len 40960 data: 18 16 17 16 14 13 14 20 19 15
08/30 Read block 81 len 32768
09/30 Write block 258 len 8192 data: 12 12
10/30 Write block 402 len 36864 data: 16 13 15 16 14 13 12 13 15
11/30 Write block 441 len 28672 data: 12 12 12 9 12 13 13
12/30 Read block 486 len 12288
13/30 Write block 572 len 28672 data: 16 17 16 19 16 16 13
14/30 Write block 284 len 12288 data: 12 14 11
15/30 Read block 491 len 20480
16/30 Read block 56 len 28672
17/30 Write block 65 len 16384 data: 21 20 19 21
18/30 Read block 442 len 8192
19/30 Write block 321 len 24576 data: 17 18 20 21 23 22
20/30 Write block 369 len 4096 data: 18
21/30 Write block 567 len 32768 data: 12 11 12 15 15 17 18 17
22/30 Write block 427 len 20480 data: 13 10 11 13 13
23/30 Write block 407 len 12288 data: 14 13 14
24/30 Read block 229 len 24576
25/30 Read block 413 len 32768
26/30 Flush
27/30 Write block 489 len 8192 data: 14 16
28/30 Write block 39 len 40960 data: 12 14 16 13 13 12 12 12 10 10
29/30 Write block 310 len 40960 data: 13 13 12 13 16 15 14 13 11 16
30/30 Write block 526 len 24576 data: 14 15 17 19 18 18
----------------------------------------------------------------
Crucible gen:198 GIO:true work queues: Upstairs:0 downstairs:4
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1026 Acked Write 2 Done Done Done false
1027 Acked Write 10 Done Done Done false
1028 Acked Write 10 Sent Sent Sent false
1029 Acked Write 6 Sent Sent Sent false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 2 2 2 6
Done 2 2 2 6
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1025 1025 1025
Downstairs last five completed: 1025 1024 1023 1022 1021
Upstairs last five completed: 1029 1028 1027 1026 1025
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-27T19:44:55.309486138Z","hostname":"w-01JJMM4YQC45QTGBDP0DKQX50W","pid":1974}
Wrote out file "/tmp/test_repair_verify.data"
Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 191 191 191 347 347 347 F F F
1 020-039 193 193 193 349 349 349 T T T
2 040-059 194 194 194 351 351 351 T T T
3 060-079 198 198 198 359 359 359 F F F
4 080-099 197 197 197 357 357 357 F F F
5 100-119 197 197 197 357 357 357 F F F
6 120-139 191 191 191 347 347 347 F F F
7 140-159 198 198 198 359 359 359 F F F
8 160-179 196 196 196 355 355 355 F F F
9 180-199 198 198 198 358 358 358 F F F
10 200-219 198 198 198 358 358 358 F F F
11 220-239 196 196 196 356 356 356 F F F
12 240-259 198 198 198 359 359 359 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 198 198 198 359 359 359 F F F
15 300-319 194 194 194 351 351 351 T T T
16 320-339 198 198 198 359 359 359 F F F
17 340-359 191 191 191 347 347 347 F F F
18 360-379 198 198 198 359 359 359 F F F
19 380-399 192 192 192 348 348 348 F F F
20 400-419 198 198 198 359 359 359 F F F
21 420-439 198 198 198 359 359 359 F F F
22 440-459 198 198 198 359 359 359 F F F
23 460-479 198 198 198 359 359 359 F F F
24 480-499 192 192 192 348 348 348 T T T
25 500-519 197 197 197 357 357 357 F F F
26 520-539 197 197 197 357 357 357 T T T
27 540-559 194 194 194 351 351 351 F F F
28 560-579 198 198 198 359 359 359 F F F
29 580-599 197 197 197 357 357 357 F F F
Max gen: 198, Max flush: 359
A Difference in extent metadata is expected here
On loop 99
Verifying data now
No change after verify
Loop: 99 Downstairs dump after verify (and repair):
EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF
0 000-019 191 191 191 347 347 347 F F F
1 020-039 199 199 199 360 360 360 F F F
2 040-059 199 199 199 360 360 360 F F F
3 060-079 198 198 198 359 359 359 F F F
4 080-099 197 197 197 357 357 357 F F F
5 100-119 197 197 197 357 357 357 F F F
6 120-139 191 191 191 347 347 347 F F F
7 140-159 198 198 198 359 359 359 F F F
8 160-179 196 196 196 355 355 355 F F F
9 180-199 198 198 198 358 358 358 F F F
10 200-219 198 198 198 358 358 358 F F F
11 220-239 196 196 196 356 356 356 F F F
12 240-259 198 198 198 359 359 359 F F F
13 260-279 172 172 172 315 315 315 F F F
14 280-299 198 198 198 359 359 359 F F F
15 300-319 199 199 199 360 360 360 F F F
16 320-339 198 198 198 359 359 359 F F F
17 340-359 191 191 191 347 347 347 F F F
18 360-379 198 198 198 359 359 359 F F F
19 380-399 192 192 192 348 348 348 F F F
20 400-419 198 198 198 359 359 359 F F F
21 420-439 198 198 198 359 359 359 F F F
22 440-459 198 198 198 359 359 359 F F F
23 460-479 198 198 198 359 359 359 F F F
24 480-499 199 199 199 360 360 360 F F F
25 500-519 197 197 197 357 357 357 F F F
26 520-539 199 199 199 360 360 360 F F F
27 540-559 194 194 194 351 351 351 F F F
28 560-579 198 198 198 359 359 359 F F F
29 580-599 197 197 197 357 357 357 F F F
Max gen: 199, Max flush: 360
8:46 Test duration
Test completed
real 8:45.867686213
user 49.847520342
sys 20.369827276
trap 0.073218584
tflt 0.083947408
dflt 0.145759703
kflt 0.007152010
lock 14:04:22.506373167
slp 42:52.143929442
lat 5.948871652
stop 0.066923721