/var/tmp/bins/crutest verify -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 --verify-out /tmp/test_repair_verify.data --verify-in /tmp/test_repair_verify.data --range -q -g 199 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.33004556Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-21T15:55:01.373177035Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.373637537Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.404685386Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.40530384Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"5c2b227300b758c153f33da870809d67a5160dae\",\n git_commit_timestamp: \"2025-01-21T15:31:20.000000000Z\",\n git_branch: \"mkeeter/always-check-live-repair\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.405337978Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980} {"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.405352476Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.405366704Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980} {"msg":"Crucible f9643ad1-76ed-46dc-b241-9e8bc153a31e has session id: 6080cee6-abd3-4b5d-8d2d-9d55e783fe0f","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.405979445Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Upstairs opts: Upstairs UUID: f9643ad1-76ed-46dc-b241-9e8bc153a31e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406010711Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Crucible stats registered with UUID: f9643ad1-76ed-46dc-b241-9e8bc153a31e","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406024878Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-21T15:55:01.406037165Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406448341Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"task":"crutest"} {"msg":"The guest has requested activation with gen:199","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406472303Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980} {"msg":"f9643ad1-76ed-46dc-b241-9e8bc153a31e active request set","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406537547Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406555127Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"0","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.40658009Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"0","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406604673Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406632337Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406660952Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406688417Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406715991Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"io task","client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406743305Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"io task","client":"0","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.40677092Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"io task","client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"ds_connection connected from Ok(127.0.0.1:44328)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406797524Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"io task","client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"ds_connection connected from Ok(127.0.0.1:35369)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406967443Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"io task","client":"0","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"ds_connection connected from Ok(127.0.0.1:44173)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.406994497Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"io task","client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.407538552Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.407574011Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.407607218Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"0","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.408185972Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.408344025Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"0","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.40837381Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.408847398Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.408949032Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"[0]R flush_numbers[0..12]: [320, 315, 319, 322, 317, 318, 324, 318, 323, 323, 323, 313]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409105544Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"[0]R generation[0..12]: [195, 192, 194, 196, 193, 194, 198, 194, 197, 197, 197, 191]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409126115Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"[0]R dirty[0..12]: [false, true, true, false, false, false, true, false, true, false, true, true]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409140522Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"[1]R flush_numbers[0..12]: [320, 315, 319, 322, 317, 318, 324, 318, 323, 323, 323, 313]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.40915443Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"[1]R generation[0..12]: [195, 192, 194, 196, 193, 194, 198, 194, 197, 197, 197, 191]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409167937Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"[1]R dirty[0..12]: [false, true, true, false, false, false, true, false, true, false, true, true]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409181624Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"[2]R flush_numbers[0..12]: [320, 315, 319, 322, 317, 318, 324, 318, 323, 323, 323, 313]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409195101Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"[2]R generation[0..12]: [195, 192, 194, 196, 193, 194, 198, 194, 197, 197, 197, 191]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409211429Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"[2]R dirty[0..12]: [false, true, true, false, false, false, true, false, true, false, true, true]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409225047Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Max found gen is 199","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409237763Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Generation requested: 199 >= found:199","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.40925083Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Next flush: 325","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409264587Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent 1 needs reconciliation: ExtentMetadata { gen: 192, flush: 315, dirty: true } ExtentMetadata { gen: 192, flush: 315, dirty: true } ExtentMetadata { gen: 192, flush: 315, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409279965Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent:0 [ExtentMetadata { gen: 192, flush: 315, dirty: true }, ExtentMetadata { gen: 192, flush: 315, dirty: true }, ExtentMetadata { gen: 192, flush: 315, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409295804Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"found dest for source 0 for extent at index 1 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409309821Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent 2 needs reconciliation: ExtentMetadata { gen: 194, flush: 319, dirty: true } ExtentMetadata { gen: 194, flush: 319, dirty: true } ExtentMetadata { gen: 194, flush: 319, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409324949Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent:0 [ExtentMetadata { gen: 194, flush: 319, dirty: true }, ExtentMetadata { gen: 194, flush: 319, dirty: true }, ExtentMetadata { gen: 194, flush: 319, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409340027Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"found dest for source 0 for extent at index 2 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409354064Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent 6 needs reconciliation: ExtentMetadata { gen: 198, flush: 324, dirty: true } ExtentMetadata { gen: 198, flush: 324, dirty: true } ExtentMetadata { gen: 198, flush: 324, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409369192Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent:0 [ExtentMetadata { gen: 198, flush: 324, dirty: true }, ExtentMetadata { gen: 198, flush: 324, dirty: true }, ExtentMetadata { gen: 198, flush: 324, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409386171Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"found dest for source 0 for extent at index 6 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409399908Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent 8 needs reconciliation: ExtentMetadata { gen: 197, flush: 323, dirty: true } ExtentMetadata { gen: 197, flush: 323, dirty: true } ExtentMetadata { gen: 197, flush: 323, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409415246Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent:0 [ExtentMetadata { gen: 197, flush: 323, dirty: true }, ExtentMetadata { gen: 197, flush: 323, dirty: true }, ExtentMetadata { gen: 197, flush: 323, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409430324Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"found dest for source 0 for extent at index 8 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409443901Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent 10 needs reconciliation: ExtentMetadata { gen: 197, flush: 323, dirty: true } ExtentMetadata { gen: 197, flush: 323, dirty: true } ExtentMetadata { gen: 197, flush: 323, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409459159Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent:0 [ExtentMetadata { gen: 197, flush: 323, dirty: true }, ExtentMetadata { gen: 197, flush: 323, dirty: true }, ExtentMetadata { gen: 197, flush: 323, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409474017Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"found dest for source 0 for extent at index 10 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409487754Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent 11 needs reconciliation: ExtentMetadata { gen: 191, flush: 313, dirty: true } ExtentMetadata { gen: 191, flush: 313, dirty: true } ExtentMetadata { gen: 191, flush: 313, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409502782Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent:0 [ExtentMetadata { gen: 191, flush: 313, dirty: true }, ExtentMetadata { gen: 191, flush: 313, dirty: true }, ExtentMetadata { gen: 191, flush: 313, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.4095176Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"found dest for source 0 for extent at index 11 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409531397Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent 13 needs reconciliation: ExtentMetadata { gen: 197, flush: 323, dirty: true } ExtentMetadata { gen: 197, flush: 323, dirty: true } ExtentMetadata { gen: 197, flush: 323, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409548176Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent:0 [ExtentMetadata { gen: 197, flush: 323, dirty: true }, ExtentMetadata { gen: 197, flush: 323, dirty: true }, ExtentMetadata { gen: 197, flush: 323, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409563314Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"found dest for source 0 for extent at index 13 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409576951Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent 17 needs reconciliation: ExtentMetadata { gen: 195, flush: 320, dirty: true } ExtentMetadata { gen: 195, flush: 320, dirty: true } ExtentMetadata { gen: 195, flush: 320, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409592119Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent:0 [ExtentMetadata { gen: 195, flush: 320, dirty: true }, ExtentMetadata { gen: 195, flush: 320, dirty: true }, ExtentMetadata { gen: 195, flush: 320, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409607157Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"found dest for source 0 for extent at index 17 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409620754Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent 18 needs reconciliation: ExtentMetadata { gen: 197, flush: 323, dirty: true } ExtentMetadata { gen: 197, flush: 323, dirty: true } ExtentMetadata { gen: 197, flush: 323, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409636062Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent:0 [ExtentMetadata { gen: 197, flush: 323, dirty: true }, ExtentMetadata { gen: 197, flush: 323, dirty: true }, ExtentMetadata { gen: 197, flush: 323, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.40965109Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"found dest for source 0 for extent at index 18 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409664977Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent 20 needs reconciliation: ExtentMetadata { gen: 192, flush: 315, dirty: true } ExtentMetadata { gen: 192, flush: 315, dirty: true } ExtentMetadata { gen: 192, flush: 315, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409680065Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent:0 [ExtentMetadata { gen: 192, flush: 315, dirty: true }, ExtentMetadata { gen: 192, flush: 315, dirty: true }, ExtentMetadata { gen: 192, flush: 315, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409695093Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"found dest for source 0 for extent at index 20 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409710161Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent 24 needs reconciliation: ExtentMetadata { gen: 197, flush: 323, dirty: true } ExtentMetadata { gen: 197, flush: 323, dirty: true } ExtentMetadata { gen: 197, flush: 323, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409725559Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent:0 [ExtentMetadata { gen: 197, flush: 323, dirty: true }, ExtentMetadata { gen: 197, flush: 323, dirty: true }, ExtentMetadata { gen: 197, flush: 323, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409740447Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"found dest for source 0 for extent at index 24 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409754004Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent 25 needs reconciliation: ExtentMetadata { gen: 192, flush: 316, dirty: true } ExtentMetadata { gen: 192, flush: 316, dirty: true } ExtentMetadata { gen: 192, flush: 316, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409769092Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent:0 [ExtentMetadata { gen: 192, flush: 316, dirty: true }, ExtentMetadata { gen: 192, flush: 316, dirty: true }, ExtentMetadata { gen: 192, flush: 316, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.40978401Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"found dest for source 0 for extent at index 25 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409797707Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent 26 needs reconciliation: ExtentMetadata { gen: 190, flush: 311, dirty: true } ExtentMetadata { gen: 190, flush: 311, dirty: true } ExtentMetadata { gen: 190, flush: 311, dirty: true }","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.40982208Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"extent:0 [ExtentMetadata { gen: 190, flush: 311, dirty: true }, ExtentMetadata { gen: 190, flush: 311, dirty: true }, ExtentMetadata { gen: 190, flush: 311, dirty: true }] => 0","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409838848Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"found dest for source 0 for extent at index 26 => [ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409852726Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409873207Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"0","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409889765Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409903362Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Full repair list: {ExtentId(25): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(18): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(26): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(2): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(6): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(8): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(13): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(17): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(11): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(10): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(24): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(1): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(20): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }}","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.409931097Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Task list: [ReconcileIO { id: ReconciliationId(0), op: ExtentFlush { repair_id: ReconciliationId(0), extent_id: ExtentId(25), client_id: ClientId(0), flush_number: 325, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(1), op: ExtentClose { repair_id: ReconciliationId(1), extent_id: ExtentId(25) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(2), op: ExtentRepair { repair_id: ReconciliationId(2), extent_id: ExtentId(25), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(3), op: ExtentReopen { repair_id: ReconciliationId(3), extent_id: ExtentId(25) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(4), op: ExtentFlush { repair_id: ReconciliationId(4), extent_id: ExtentId(18), client_id: ClientId(0), flush_number: 325, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(5), op: ExtentClose { repair_id: ReconciliationId(5), extent_id: ExtentId(18) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(6), op: ExtentRepair { repair_id: ReconciliationId(6), extent_id: ExtentId(18), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(7), op: ExtentReopen { repair_id: ReconciliationId(7), extent_id: ExtentId(18) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(8), op: ExtentFlush { repair_id: ReconciliationId(8), extent_id: ExtentId(26), client_id: ClientId(0), flush_number: 325, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(9), op: ExtentClose { repair_id: ReconciliationId(9), extent_id: ExtentId(26) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(10), op: ExtentRepair { repair_id: ReconciliationId(10), extent_id: ExtentId(26), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(11), op: ExtentReopen { repair_id: ReconciliationId(11), extent_id: ExtentId(26) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(12), op: ExtentFlush { repair_id: ReconciliationId(12), extent_id: ExtentId(2), client_id: ClientId(0), flush_number: 325, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(13), op: ExtentClose { repair_id: ReconciliationId(13), extent_id: ExtentId(2) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(14), op: ExtentRepair { repair_id: ReconciliationId(14), extent_id: ExtentId(2), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(15), op: ExtentReopen { repair_id: ReconciliationId(15), extent_id: ExtentId(2) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(16), op: ExtentFlush { repair_id: ReconciliationId(16), extent_id: ExtentId(6), client_id: ClientId(0), flush_number: 325, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(17), op: ExtentClose { repair_id: ReconciliationId(17), extent_id: ExtentId(6) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(18), op: ExtentRepair { repair_id: ReconciliationId(18), extent_id: ExtentId(6), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(19), op: ExtentReopen { repair_id: ReconciliationId(19), extent_id: ExtentId(6) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(20), op: ExtentFlush { repair_id: ReconciliationId(20), extent_id: ExtentId(8), client_id: ClientId(0), flush_number: 325, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(21), op: ExtentClose { repair_id: ReconciliationId(21), extent_id: ExtentId(8) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(22), op: ExtentRepair { repair_id: ReconciliationId(22), extent_id: ExtentId(8), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(23), op: ExtentReopen { repair_id: ReconciliationId(23), extent_id: ExtentId(8) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(24), op: ExtentFlush { repair_id: ReconciliationId(24), extent_id: ExtentId(13), client_id: ClientId(0), flush_number: 325, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(25), op: ExtentClose { repair_id: ReconciliationId(25), extent_id: ExtentId(13) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(26), op: ExtentRepair { repair_id: ReconciliationId(26), extent_id: ExtentId(13), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(27), op: ExtentReopen { repair_id: ReconciliationId(27), extent_id: ExtentId(13) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(28), op: ExtentFlush { repair_id: ReconciliationId(28), extent_id: ExtentId(17), client_id: ClientId(0), flush_number: 325, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(29), op: ExtentClose { repair_id: ReconciliationId(29), extent_id: ExtentId(17) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(30), op: ExtentRepair { repair_id: ReconciliationId(30), extent_id: ExtentId(17), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(31), op: ExtentReopen { repair_id: ReconciliationId(31), extent_id: ExtentId(17) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(32), op: ExtentFlush { repair_id: ReconciliationId(32), extent_id: ExtentId(11), client_id: ClientId(0), flush_number: 325, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(33), op: ExtentClose { repair_id: ReconciliationId(33), extent_id: ExtentId(11) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(34), op: ExtentRepair { repair_id: ReconciliationId(34), extent_id: ExtentId(11), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(35), op: ExtentReopen { repair_id: ReconciliationId(35), extent_id: ExtentId(11) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(36), op: ExtentFlush { repair_id: ReconciliationId(36), extent_id: ExtentId(10), client_id: ClientId(0), flush_number: 325, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(37), op: ExtentClose { repair_id: ReconciliationId(37), extent_id: ExtentId(10) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(38), op: ExtentRepair { repair_id: ReconciliationId(38), extent_id: ExtentId(10), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(39), op: ExtentReopen { repair_id: ReconciliationId(39), extent_id: ExtentId(10) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(40), op: ExtentFlush { repair_id: ReconciliationId(40), extent_id: ExtentId(24), client_id: ClientId(0), flush_number: 325, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(41), op: ExtentClose { repair_id: ReconciliationId(41), extent_id: ExtentId(24) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(42), op: ExtentRepair { repair_id: ReconciliationId(42), extent_id: ExtentId(24), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(43), op: ExtentReopen { repair_id: ReconciliationId(43), extent_id: ExtentId(24) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(44), op: ExtentFlush { repair_id: ReconciliationId(44), extent_id: ExtentId(1), client_id: ClientId(0), flush_number: 325, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(45), op: ExtentClose { repair_id: ReconciliationId(45), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(46), op: ExtentRepair { repair_id: ReconciliationId(46), extent_id: ExtentId(1), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(47), op: ExtentReopen { repair_id: ReconciliationId(47), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(48), op: ExtentFlush { repair_id: ReconciliationId(48), extent_id: ExtentId(20), client_id: ClientId(0), flush_number: 325, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(49), op: ExtentClose { repair_id: ReconciliationId(49), extent_id: ExtentId(20) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(50), op: ExtentRepair { repair_id: ReconciliationId(50), extent_id: ExtentId(20), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(51), op: ExtentReopen { repair_id: ReconciliationId(51), extent_id: ExtentId(20) }, state: ClientData([New, New, New]) }]","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.41008991Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"starting reconciliation 870208d4-02fe-4b04-ae56-6aa0ba42b59e: found 52 extents that need repair","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.410105498Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.410118945Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.410133303Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.415818844Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.41584979Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.420384768Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.420414243Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.424747375Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.424776591Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.42931445Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.429343325Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(22)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.433606291Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(22)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.433635596Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(26)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.437725491Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(26)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.437760809Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(30)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.44407631Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(30)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.444114911Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(34)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.448470665Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(34)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.448508434Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(38)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.452729678Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(38)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.452757822Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(42)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.457056146Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(42)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.457084991Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(46)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.463616606Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(46)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.463646972Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(50)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.467996123Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"sending reconcile request ReconciliationId(50)","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.468025468Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"done with reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.471706108Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"":"downstairs","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.471735373Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"0","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.471751472Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"1","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.471769491Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"client":"2","":"downstairs","session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.471783008Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.471795304Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"f9643ad1-76ed-46dc-b241-9e8bc153a31e is now active with session: 6080cee6-abd3-4b5d-8d2d-9d55e783fe0f","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.471808181Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980,"session_id":"6080cee6-abd3-4b5d-8d2d-9d55e783fe0f"} {"msg":"The guest has finished waiting for activation with:199","v":0,"name":"crucible","level":30,"time":"2025-01-21T15:55:01.471820538Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Read and Verify all blocks (0..600 range:true) Wrote out file "/tmp/test_repair_verify.data" Verify test completed Wrote out file "/tmp/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Read 88 Done Done Done false 1001 Acked Read 256 Done Done Done false 1002 Acked Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 0 0 0 Downstairs last five completed: Upstairs last five completed: 1002 1001 1000 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1003 1002 1001 1000 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-21T15:55:05.526993921Z","hostname":"w-01JJ4RN6KGTRRMQBMR2CC9MVGG","pid":1980}