/var/tmp/bins/crutest verify -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 --verify-out /tmp/test_repair_verify.data --verify-in /tmp/test_repair_verify.data --range -q -g 199 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.053209859Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.098659505Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.099403517Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"008b9b7890522309c5657fa001c5655325f06c1f\",\n git_commit_timestamp: \"2025-01-15T05:03:03.000000000Z\",\n git_branch: \"renovate/http-1.x\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.099447327Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980} {"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.099463223Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.09947756Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980} {"msg":"Crucible b0234de0-81e3-4004-8947-42242bc7e864 has session id: 1118c72b-9607-4937-be94-47524f68d7c8","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100134641Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Upstairs opts: Upstairs UUID: b0234de0-81e3-4004-8947-42242bc7e864, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100180541Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Crucible stats registered with UUID: b0234de0-81e3-4004-8947-42242bc7e864","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100204795Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-15T05:26:52.10022709Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100619302Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"task":"crutest"} {"msg":"The guest has requested activation with gen:199","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100645036Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980} {"msg":"b0234de0-81e3-4004-8947-42242bc7e864 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100721148Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100747713Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"0","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100777836Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"0","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100795272Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100823255Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100849059Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100875224Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100904827Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"io task","client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.10093377Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"io task","client":"0","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100958285Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"io task","client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"ds_connection connected from Ok(127.0.0.1:47462)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.100986718Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"io task","client":"0","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"ds_connection connected from Ok(127.0.0.1:34661)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.101013392Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"io task","client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"ds_connection connected from Ok(127.0.0.1:44708)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.101162019Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"io task","client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.10181765Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.102005058Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.102080261Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"0","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.1025237Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.102549804Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"0","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.102664978Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103088092Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103204496Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"[0]R flush_numbers[0..12]: [346, 347, 328, 327, 344, 331, 335, 348, 348, 343, 319, 345]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103269951Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"[0]R generation[0..12]: [197, 198, 184, 184, 195, 188, 190, 198, 198, 194, 179, 196]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103299095Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, true, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103315401Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"[1]R flush_numbers[0..12]: [346, 347, 328, 327, 344, 331, 335, 348, 348, 343, 319, 345]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103329688Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"[1]R generation[0..12]: [197, 198, 184, 184, 195, 188, 190, 198, 198, 194, 179, 196]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103343374Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, true, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103357131Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"[2]R flush_numbers[0..12]: [346, 347, 328, 327, 344, 331, 335, 348, 348, 343, 319, 345]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103370988Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"[2]R generation[0..12]: [197, 198, 184, 184, 195, 188, 190, 198, 198, 194, 179, 196]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103384665Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, true, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103398462Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Max found gen is 199","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103415518Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Generation requested: 199 >= found:199","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103429055Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Next flush: 349","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103442012Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Extents 9 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103454859Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"First source client ID for extent 9","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103468116Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:9 gens: 194 194 194","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103481973Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:9 flush: 343 343 343 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.10349631Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103510637Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:9 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103524484Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"find dest for source 0 for extent at index 9","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.10353839Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103551937Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103565794Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Extents 14 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.10358573Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"First source client ID for extent 14","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103599747Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:14 gens: 197 197 197","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103623831Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:14 flush: 346 346 346 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103664442Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103693815Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:14 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103723188Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"find dest for source 0 for extent at index 14","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103751552Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103775137Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103799071Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Extents 16 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103824736Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"First source client ID for extent 16","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.10384846Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:16 gens: 192 192 192","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103872475Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:16 flush: 340 340 340 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103897679Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103923303Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:16 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103951967Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"find dest for source 0 for extent at index 16","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103979881Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.103999296Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104017132Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Extents 18 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104031769Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"First source client ID for extent 18","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104045875Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:18 gens: 195 195 195","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104059672Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:18 flush: 344 344 344 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104073939Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104095544Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:18 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104122448Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"find dest for source 0 for extent at index 18","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104150072Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104180945Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104211318Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Extents 19 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104240631Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"First source client ID for extent 19","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104256908Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:19 gens: 198 198 198","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104270915Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:19 flush: 347 347 347 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104286361Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104303317Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:19 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104317164Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"find dest for source 0 for extent at index 19","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104331281Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104345068Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104358984Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Extents 27 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104372721Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"First source client ID for extent 27","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104385929Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:27 gens: 196 196 196","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104399665Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:27 flush: 345 345 345 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104413902Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104427919Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:27 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104441796Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"find dest for source 0 for extent at index 27","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104455613Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104469399Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104483087Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Extents 28 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104496763Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"First source client ID for extent 28","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.10451195Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:28 gens: 195 195 195","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104526027Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:28 flush: 344 344 344 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104540274Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.10455439Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:28 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104568177Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"find dest for source 0 for extent at index 28","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104582124Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104595951Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104609688Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Extents 29 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104623205Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"First source client ID for extent 29","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104636392Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:29 gens: 190 190 190","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104650039Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:29 flush: 337 337 337 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104664255Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104678402Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"extent:29 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104692259Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"find dest for source 0 for extent at index 29","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104707956Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104721822Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104735619Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104764813Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"0","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.1047789Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104801654Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Full repair list: {ExtentId(9): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(14): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(28): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(29): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(16): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(19): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(27): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(18): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }}","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.104841485Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Task list: [ReconcileIO { id: ReconciliationId(0), op: ExtentFlush { repair_id: ReconciliationId(0), extent_id: ExtentId(9), client_id: ClientId(0), flush_number: 349, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(1), op: ExtentClose { repair_id: ReconciliationId(1), extent_id: ExtentId(9) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(2), op: ExtentRepair { repair_id: ReconciliationId(2), extent_id: ExtentId(9), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(3), op: ExtentReopen { repair_id: ReconciliationId(3), extent_id: ExtentId(9) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(4), op: ExtentFlush { repair_id: ReconciliationId(4), extent_id: ExtentId(14), client_id: ClientId(0), flush_number: 349, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(5), op: ExtentClose { repair_id: ReconciliationId(5), extent_id: ExtentId(14) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(6), op: ExtentRepair { repair_id: ReconciliationId(6), extent_id: ExtentId(14), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(7), op: ExtentReopen { repair_id: ReconciliationId(7), extent_id: ExtentId(14) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(8), op: ExtentFlush { repair_id: ReconciliationId(8), extent_id: ExtentId(28), client_id: ClientId(0), flush_number: 349, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(9), op: ExtentClose { repair_id: ReconciliationId(9), extent_id: ExtentId(28) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(10), op: ExtentRepair { repair_id: ReconciliationId(10), extent_id: ExtentId(28), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(11), op: ExtentReopen { repair_id: ReconciliationId(11), extent_id: ExtentId(28) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(12), op: ExtentFlush { repair_id: ReconciliationId(12), extent_id: ExtentId(29), client_id: ClientId(0), flush_number: 349, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(13), op: ExtentClose { repair_id: ReconciliationId(13), extent_id: ExtentId(29) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(14), op: ExtentRepair { repair_id: ReconciliationId(14), extent_id: ExtentId(29), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(15), op: ExtentReopen { repair_id: ReconciliationId(15), extent_id: ExtentId(29) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(16), op: ExtentFlush { repair_id: ReconciliationId(16), extent_id: ExtentId(16), client_id: ClientId(0), flush_number: 349, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(17), op: ExtentClose { repair_id: ReconciliationId(17), extent_id: ExtentId(16) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(18), op: ExtentRepair { repair_id: ReconciliationId(18), extent_id: ExtentId(16), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(19), op: ExtentReopen { repair_id: ReconciliationId(19), extent_id: ExtentId(16) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(20), op: ExtentFlush { repair_id: ReconciliationId(20), extent_id: ExtentId(19), client_id: ClientId(0), flush_number: 349, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(21), op: ExtentClose { repair_id: ReconciliationId(21), extent_id: ExtentId(19) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(22), op: ExtentRepair { repair_id: ReconciliationId(22), extent_id: ExtentId(19), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(23), op: ExtentReopen { repair_id: ReconciliationId(23), extent_id: ExtentId(19) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(24), op: ExtentFlush { repair_id: ReconciliationId(24), extent_id: ExtentId(27), client_id: ClientId(0), flush_number: 349, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(25), op: ExtentClose { repair_id: ReconciliationId(25), extent_id: ExtentId(27) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(26), op: ExtentRepair { repair_id: ReconciliationId(26), extent_id: ExtentId(27), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(27), op: ExtentReopen { repair_id: ReconciliationId(27), extent_id: ExtentId(27) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(28), op: ExtentFlush { repair_id: ReconciliationId(28), extent_id: ExtentId(18), client_id: ClientId(0), flush_number: 349, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(29), op: ExtentClose { repair_id: ReconciliationId(29), extent_id: ExtentId(18) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(30), op: ExtentRepair { repair_id: ReconciliationId(30), extent_id: ExtentId(18), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(31), op: ExtentReopen { repair_id: ReconciliationId(31), extent_id: ExtentId(18) }, state: ClientData([New, New, New]) }]","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.105029213Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"starting reconciliation 7171874d-fb31-477a-95a1-51c6ebac276f: found 32 extents that need repair","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.105068914Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.105098617Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.105125821Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.109974083Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.110002966Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.114443771Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.114472874Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.121095264Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.121123298Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.125317138Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.12535331Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(22)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.129418459Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(22)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.129460919Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(26)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.133906203Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(26)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.133935246Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(30)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.13811149Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"sending reconcile request ReconciliationId(30)","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.138143743Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"done with reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.146406712Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"":"downstairs","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.146435375Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"0","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.146452141Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"1","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.146465768Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"client":"2","":"downstairs","session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.146478845Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.146491102Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} {"msg":"b0234de0-81e3-4004-8947-42242bc7e864 is now active with session: 1118c72b-9607-4937-be94-47524f68d7c8","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.146504199Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980,"session_id":"1118c72b-9607-4937-be94-47524f68d7c8"} Wait for a query_work_queue command to finish before sending IO {"msg":"The guest has finished waiting for activation with:199","v":0,"name":"crucible","level":30,"time":"2025-01-15T05:26:52.146524115Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Read and Verify all blocks (0..600 range:true) Wrote out file "/tmp/test_repair_verify.data" Verify test completed Wrote out file "/tmp/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Read 88 Done Done Done false 1001 Acked Read 256 Done Done Done false 1002 Acked Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 0 0 0 Downstairs last five completed: Upstairs last five completed: 1002 1001 1000 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1003 1002 1001 1000 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2025-01-15T05:26:56.201356831Z","hostname":"w-01JHM6ARD39SFM9SXVPVHKAJ0E","pid":1980}