/var/tmp/bins/crutest verify -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 --verify-out /tmp/test_repair_verify.data --verify-in /tmp/test_repair_verify.data --range -q -g 199
{"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.233121116Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"task":"crutest"}
{"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:52:59.274795886Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"task":"crutest"}
{"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.275185562Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"task":"crutest"}
{"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.305838057Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"task":"crutest"}
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.306395625Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"7ae555c374784c668297ef5478b8ff4aaa8d57b4\",\n git_commit_timestamp: \"2025-02-21T06:29:49.000000000Z\",\n git_branch: \"renovate/rust-1.x\",\n rustc_semver: \"1.85.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"4d91de4e48198da2e33413efdcd9cd2cc0c46688\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.306428699Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989}
{"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.306442127Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989}
{"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.306456304Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989}
{"msg":"Crucible a8a822f7-6eea-49ba-8a6c-696c3d25601d has session id: 6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307001784Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Upstairs opts: Upstairs UUID: a8a822f7-6eea-49ba-8a6c-696c3d25601d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307044107Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Crucible stats registered with UUID: a8a822f7-6eea-49ba-8a6c-696c3d25601d","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307068143Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:52:59.307090439Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307513919Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"task":"crutest"}
{"msg":"The guest has requested activation with gen:199","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307536136Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989}
{"msg":"a8a822f7-6eea-49ba-8a6c-696c3d25601d active request set","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307597846Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307624211Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"0","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307646797Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"0","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307668964Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"1","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307695559Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"1","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307721315Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"2","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.30774891Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"2","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307773616Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"io task","client":"2","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307801952Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"io task","client":"0","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307829017Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"io task","client":"1","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"ds_connection connected from Ok(127.0.0.1:41764)","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.307856813Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"io task","client":"2","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"ds_connection connected from Ok(127.0.0.1:47049)","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.308046641Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"io task","client":"1","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"ds_connection connected from Ok(127.0.0.1:38952)","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.308078576Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"io task","client":"0","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.308632444Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"1","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.308699313Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"2","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.308745805Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"0","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.309297814Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"2","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.309334348Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"0","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.309362423Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"1","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.309874169Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.309896755Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"[0]R flush_numbers[0..12]: [291, 313, 323, 318, 317, 320, 320, 318, 325, 322, 322, 322]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.309950846Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"[0]R generation[0..12]: [179, 190, 198, 194, 193, 195, 195, 194, 198, 197, 197, 197]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.309970383Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"[0]R dirty[0..12]: [true, true, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.30998463Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"[1]R flush_numbers[0..12]: [291, 313, 323, 318, 317, 320, 320, 318, 325, 322, 322, 322]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.309998518Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"[1]R generation[0..12]: [179, 190, 198, 194, 193, 195, 195, 194, 198, 197, 197, 197]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310012256Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"[1]R dirty[0..12]: [true, true, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310026233Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"[2]R flush_numbers[0..12]: [291, 313, 323, 318, 317, 320, 320, 318, 325, 322, 322, 322]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310039811Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"[2]R generation[0..12]: [179, 190, 198, 194, 193, 195, 195, 194, 198, 197, 197, 197]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310057068Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"[2]R dirty[0..12]: [true, true, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310071276Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Max found gen is 199","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310084524Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Generation requested: 199 >= found:199","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310097621Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Next flush: 326","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.31011031Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"extent 0 needs reconciliation: ExtentMetadata { gen: 179, flush: 291, dirty: true } ExtentMetadata { gen: 179, flush: 291, dirty: true } ExtentMetadata { gen: 179, flush: 291, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310125627Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"extent:0 [ExtentMetadata { gen: 179, flush: 291, dirty: true }, ExtentMetadata { gen: 179, flush: 291, dirty: true }, ExtentMetadata { gen: 179, flush: 291, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310141444Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"found dest for source 0 for extent at index 0 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310155492Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"extent 1 needs reconciliation: ExtentMetadata { gen: 190, flush: 313, dirty: true } ExtentMetadata { gen: 190, flush: 313, dirty: true } ExtentMetadata { gen: 190, flush: 313, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310170949Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"extent:0 [ExtentMetadata { gen: 190, flush: 313, dirty: true }, ExtentMetadata { gen: 190, flush: 313, dirty: true }, ExtentMetadata { gen: 190, flush: 313, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310185857Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"found dest for source 0 for extent at index 1 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310199685Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"extent 15 needs reconciliation: ExtentMetadata { gen: 197, flush: 322, dirty: true } ExtentMetadata { gen: 197, flush: 322, dirty: true } ExtentMetadata { gen: 197, flush: 322, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310214842Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"extent:0 [ExtentMetadata { gen: 197, flush: 322, dirty: true }, ExtentMetadata { gen: 197, flush: 322, dirty: true }, ExtentMetadata { gen: 197, flush: 322, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310231479Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"found dest for source 0 for extent at index 15 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310245547Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"extent 16 needs reconciliation: ExtentMetadata { gen: 196, flush: 321, dirty: true } ExtentMetadata { gen: 196, flush: 321, dirty: true } ExtentMetadata { gen: 196, flush: 321, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310260854Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"extent:0 [ExtentMetadata { gen: 196, flush: 321, dirty: true }, ExtentMetadata { gen: 196, flush: 321, dirty: true }, ExtentMetadata { gen: 196, flush: 321, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310275832Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"found dest for source 0 for extent at index 16 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.31028967Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"extent 21 needs reconciliation: ExtentMetadata { gen: 188, flush: 310, dirty: true } ExtentMetadata { gen: 188, flush: 310, dirty: true } ExtentMetadata { gen: 188, flush: 310, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310304807Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"extent:0 [ExtentMetadata { gen: 188, flush: 310, dirty: true }, ExtentMetadata { gen: 188, flush: 310, dirty: true }, ExtentMetadata { gen: 188, flush: 310, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310319855Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"found dest for source 0 for extent at index 21 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310333633Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"mend","":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310352849Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"0","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310366797Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"1","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310380105Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"2","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Full repair list: {ExtentId(0): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(1): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(15): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(21): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(16): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }}","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310398032Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Task list: [ReconcileIO { id: ReconciliationId(0), op: ExtentFlush { repair_id: ReconciliationId(0), extent_id: ExtentId(0), client_id: ClientId(0), flush_number: 326, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(1), op: ExtentClose { repair_id: ReconciliationId(1), extent_id: ExtentId(0) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(2), op: ExtentRepair { repair_id: ReconciliationId(2), extent_id: ExtentId(0), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(3), op: ExtentReopen { repair_id: ReconciliationId(3), extent_id: ExtentId(0) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(4), op: ExtentFlush { repair_id: ReconciliationId(4), extent_id: ExtentId(1), client_id: ClientId(0), flush_number: 326, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(5), op: ExtentClose { repair_id: ReconciliationId(5), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(6), op: ExtentRepair { repair_id: ReconciliationId(6), extent_id: ExtentId(1), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(7), op: ExtentReopen { repair_id: ReconciliationId(7), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(8), op: ExtentFlush { repair_id: ReconciliationId(8), extent_id: ExtentId(15), client_id: ClientId(0), flush_number: 326, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(9), op: ExtentClose { repair_id: ReconciliationId(9), extent_id: ExtentId(15) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(10), op: ExtentRepair { repair_id: ReconciliationId(10), extent_id: ExtentId(15), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(11), op: ExtentReopen { repair_id: ReconciliationId(11), extent_id: ExtentId(15) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(12), op: ExtentFlush { repair_id: ReconciliationId(12), extent_id: ExtentId(21), client_id: ClientId(0), flush_number: 326, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(13), op: ExtentClose { repair_id: ReconciliationId(13), extent_id: ExtentId(21) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(14), op: ExtentRepair { repair_id: ReconciliationId(14), extent_id: ExtentId(21), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(15), op: ExtentReopen { repair_id: ReconciliationId(15), extent_id: ExtentId(21) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(16), op: ExtentFlush { repair_id: ReconciliationId(16), extent_id: ExtentId(16), client_id: ClientId(0), flush_number: 326, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(17), op: ExtentClose { repair_id: ReconciliationId(17), extent_id: ExtentId(16) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(18), op: ExtentRepair { repair_id: ReconciliationId(18), extent_id: ExtentId(16), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(19), op: ExtentReopen { repair_id: ReconciliationId(19), extent_id: ExtentId(16) }, state: ClientData([New, New, New]) }]","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.31046853Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"starting reconciliation bdf1fcbe-ca34-4ebb-a929-8791003143ea: found 20 extents that need repair","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310484828Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310819492Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"1","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.310840938Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"2","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.321525429Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"1","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.321572682Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"2","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.325976063Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"1","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.326004178Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"2","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.330290658Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"1","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.330324263Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"2","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.334525487Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"1","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.334553612Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"2","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"done with reconciliation","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.33837554Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"":"downstairs","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.338402785Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"0","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.338418632Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"1","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.3384362Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"client":"2","":"downstairs","session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.338455937Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
Wait for a query_work_queue command to finish before sending IO
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.338474983Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"a8a822f7-6eea-49ba-8a6c-696c3d25601d is now active with session: 6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.338487931Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989,"session_id":"6ecd6885-52cb-4dd5-8af5-27e5d6f2ed13"}
{"msg":"The guest has finished waiting for activation with:199","v":0,"name":"crucible","level":30,"time":"2025-02-21T06:52:59.338513227Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989}
Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576
Loading write count information from file "/tmp/test_repair_verify.data"
Read and Verify all blocks (0..600 range:true)
Wrote out file "/tmp/test_repair_verify.data"
Verify test completed
Wrote out file "/tmp/test_repair_verify.data"
CLIENT: Tests done. All submitted work has been ACK'd
----------------------------------------------------------------
Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:3
States: Active Active Active
JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1000 Acked Read 88 Done Done Done false
1001 Acked Read 256 Done Done Done false
1002 Acked Read 256 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
Sent 0 0 0 0
Done 3 3 3 9
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: None None None
Downstairs last five completed:
Upstairs last five completed: 1002 1001 1000
CLIENT: Up:0 ds:3 act:3
----------------------------------------------------------------
Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1003 1002 1001 1000
Upstairs last five completed: 1003 1002 1001 1000
CLIENT: Up:0 ds:0 act:3
CLIENT: All crucible jobs finished, exiting program
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-02-21T06:53:03.387624469Z","hostname":"w-01JMKKZ435WR2BFDX5TYESJR88","pid":1989}