/var/tmp/bins/crutest verify -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 --verify-out /tmp/test_repair_verify.data --verify-in /tmp/test_repair_verify.data --range -q -g 199 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.421841738Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2025-01-17T21:19:37.464485021Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.464902045Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.496718441Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.497307175Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"abd5430f1508af0bc008143ea2566277ef714ec2\",\n git_commit_timestamp: \"2025-01-17T20:55:03.000000000Z\",\n git_branch: \"alan/tokio-to-140\",\n rustc_semver: \"1.84.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"9fc6b43126469e3858e2fe86cafb4f0fd5068869\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.497339834Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978} {"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.497353272Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.49736738Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978} {"msg":"Crucible 78d544f2-c9d4-4c50-a32a-00b4dd1e87d1 has session id: de8b85d5-1e0b-4654-8192-de4da93cb3ad","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.497950041Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Upstairs opts: Upstairs UUID: 78d544f2-c9d4-4c50-a32a-00b4dd1e87d1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.497996328Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Crucible stats registered with UUID: 78d544f2-c9d4-4c50-a32a-00b4dd1e87d1","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.498024114Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2025-01-17T21:19:37.498048658Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.498523316Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"task":"crutest"} {"msg":"The guest has requested activation with gen:199","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.498545089Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978} {"msg":"78d544f2-c9d4-4c50-a32a-00b4dd1e87d1 active request set","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.498617321Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.498634141Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"0","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.498657104Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"0","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.498670642Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.49868418Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.498698878Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.498723062Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.498746166Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"io task","client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.49877011Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"io task","client":"0","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.498795745Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:64204)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.498822971Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"io task","client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:52280)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.498850587Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"io task","client":"0","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"ds_connection connected from Ok(127.0.0.1:43137)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.499033614Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.499640238Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.499681042Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.499716233Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"0","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.500214584Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.500300875Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.500417793Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"0","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.500835107Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.500860612Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"[0]R flush_numbers[0..12]: [354, 354, 353, 347, 352, 350, 354, 350, 352, 345, 354, 344]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.500939928Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"[0]R generation[0..12]: [198, 198, 197, 194, 196, 196, 198, 196, 196, 194, 198, 193]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.500959429Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"[0]R dirty[0..12]: [false, true, false, false, false, false, false, false, true, false, true, true]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.500973798Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"[1]R flush_numbers[0..12]: [354, 354, 353, 347, 352, 350, 354, 350, 352, 345, 354, 344]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.500987456Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"[1]R generation[0..12]: [198, 198, 197, 194, 196, 196, 198, 196, 196, 194, 198, 193]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501000754Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"[1]R dirty[0..12]: [false, true, false, false, false, false, false, false, true, false, true, true]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501014542Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"[2]R flush_numbers[0..12]: [354, 354, 353, 347, 352, 350, 354, 350, 352, 345, 354, 344]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501028Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"[2]R generation[0..12]: [198, 198, 197, 194, 196, 196, 198, 196, 196, 194, 198, 193]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.50104584Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"[2]R dirty[0..12]: [false, true, false, false, false, false, false, false, true, false, true, true]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501060448Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Max found gen is 199","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501073496Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Generation requested: 199 >= found:199","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501086294Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Next flush: 355","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501098781Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Extents 1 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501111218Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"First source client ID for extent 1","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501124426Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:1 gens: 198 198 198","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501138144Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:1 flush: 354 354 354 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501152542Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501166701Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:1 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501180189Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"find dest for source 0 for extent at index 1","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501194037Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501207655Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501221182Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Extents 8 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501242445Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"First source client ID for extent 8","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501258464Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:8 gens: 196 196 196","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501272292Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:8 flush: 352 352 352 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501286501Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501300419Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:8 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501313907Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"find dest for source 0 for extent at index 8","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501327615Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501341193Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501354911Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Extents 10 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501368659Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"First source client ID for extent 10","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501381887Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:10 gens: 198 198 198","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501395444Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:10 flush: 354 354 354 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501409543Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501423441Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:10 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501437249Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"find dest for source 0 for extent at index 10","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501452768Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501466466Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501480184Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Extents 11 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501493602Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"First source client ID for extent 11","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501506789Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:11 gens: 193 193 193","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501520217Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:11 flush: 344 344 344 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501534185Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501548084Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:11 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501561761Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"find dest for source 0 for extent at index 11","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501585526Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501613572Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501630462Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Extents 14 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.50164438Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"First source client ID for extent 14","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501657618Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:14 gens: 197 197 197","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501671196Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:14 flush: 353 353 353 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501688416Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501708347Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:14 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501734092Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"find dest for source 0 for extent at index 14","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501761108Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501791546Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501821704Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Extents 15 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.50185071Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"First source client ID for extent 15","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501879577Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:15 gens: 195 195 195","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501908134Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:15 flush: 349 349 349 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501938632Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501968919Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:15 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.501989462Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"find dest for source 0 for extent at index 15","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.50200392Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502017738Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502034458Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Extents 23 dirty","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502048696Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"First source client ID for extent 23","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502061884Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:23 gens: 193 193 193","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502075422Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:23 flush: 344 344 344 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.50208956Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502103318Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"extent:23 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502116906Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"find dest for source 0 for extent at index 23","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502130604Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502150096Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502175731Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502202496Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"0","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502227101Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502242329Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Full repair list: {ExtentId(1): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(11): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(14): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(23): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(15): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(10): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(8): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }}","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502262541Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Task list: [ReconcileIO { id: ReconciliationId(0), op: ExtentFlush { repair_id: ReconciliationId(0), extent_id: ExtentId(1), client_id: ClientId(0), flush_number: 355, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(1), op: ExtentClose { repair_id: ReconciliationId(1), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(2), op: ExtentRepair { repair_id: ReconciliationId(2), extent_id: ExtentId(1), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(3), op: ExtentReopen { repair_id: ReconciliationId(3), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(4), op: ExtentFlush { repair_id: ReconciliationId(4), extent_id: ExtentId(11), client_id: ClientId(0), flush_number: 355, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(5), op: ExtentClose { repair_id: ReconciliationId(5), extent_id: ExtentId(11) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(6), op: ExtentRepair { repair_id: ReconciliationId(6), extent_id: ExtentId(11), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(7), op: ExtentReopen { repair_id: ReconciliationId(7), extent_id: ExtentId(11) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(8), op: ExtentFlush { repair_id: ReconciliationId(8), extent_id: ExtentId(14), client_id: ClientId(0), flush_number: 355, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(9), op: ExtentClose { repair_id: ReconciliationId(9), extent_id: ExtentId(14) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(10), op: ExtentRepair { repair_id: ReconciliationId(10), extent_id: ExtentId(14), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(11), op: ExtentReopen { repair_id: ReconciliationId(11), extent_id: ExtentId(14) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(12), op: ExtentFlush { repair_id: ReconciliationId(12), extent_id: ExtentId(23), client_id: ClientId(0), flush_number: 355, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(13), op: ExtentClose { repair_id: ReconciliationId(13), extent_id: ExtentId(23) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(14), op: ExtentRepair { repair_id: ReconciliationId(14), extent_id: ExtentId(23), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(15), op: ExtentReopen { repair_id: ReconciliationId(15), extent_id: ExtentId(23) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(16), op: ExtentFlush { repair_id: ReconciliationId(16), extent_id: ExtentId(15), client_id: ClientId(0), flush_number: 355, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(17), op: ExtentClose { repair_id: ReconciliationId(17), extent_id: ExtentId(15) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(18), op: ExtentRepair { repair_id: ReconciliationId(18), extent_id: ExtentId(15), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(19), op: ExtentReopen { repair_id: ReconciliationId(19), extent_id: ExtentId(15) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(20), op: ExtentFlush { repair_id: ReconciliationId(20), extent_id: ExtentId(10), client_id: ClientId(0), flush_number: 355, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(21), op: ExtentClose { repair_id: ReconciliationId(21), extent_id: ExtentId(10) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(22), op: ExtentRepair { repair_id: ReconciliationId(22), extent_id: ExtentId(10), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(23), op: ExtentReopen { repair_id: ReconciliationId(23), extent_id: ExtentId(10) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(24), op: ExtentFlush { repair_id: ReconciliationId(24), extent_id: ExtentId(8), client_id: ClientId(0), flush_number: 355, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(25), op: ExtentClose { repair_id: ReconciliationId(25), extent_id: ExtentId(8) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(26), op: ExtentRepair { repair_id: ReconciliationId(26), extent_id: ExtentId(8), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(27), op: ExtentReopen { repair_id: ReconciliationId(27), extent_id: ExtentId(8) }, state: ClientData([New, New, New]) }]","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502357947Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"starting reconciliation 08cc7595-1163-4c09-8916-adefe631134f: found 28 extents that need repair","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502373226Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502387444Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.502408266Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.507418935Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.507446551Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.511997611Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.512026178Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.51634085Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.516368426Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.52306503Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.523093136Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(22)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.527492197Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(22)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.527520584Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(26)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.53189455Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"sending reconcile request ReconciliationId(26)","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.531922286Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"done with reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.535944107Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"":"downstairs","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.535971323Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"0","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.535986122Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"1","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.53599956Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"client":"2","":"downstairs","session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.536013097Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.536025375Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {Wait for a query_work_queue command to finish before sending IO "msg":"78d544f2-c9d4-4c50-a32a-00b4dd1e87d1 is now active with session: de8b85d5-1e0b-4654-8192-de4da93cb3ad","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.536044636Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978,"session_id":"de8b85d5-1e0b-4654-8192-de4da93cb3ad"} {"msg":"The guest has finished waiting for activation with:199","v":0,"name":"crucible","level":30,"time":"2025-01-17T21:19:37.536056663Z","hostname":"w-01JHV1KHN49PGZ24K9A4XN75MW","pid":1978} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Read and Verify all blocks (0..600 range:true) Wrote out file "/tmp/test_repair_verify.data" Verify test completed Wrote out file "/tmp/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Read 88 Done Done Done false 1001 Acked Read 256 Done Done Done false 1002 Acked Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 0 0 0 Downstairs last five completed: Upstairs last five completed: 1002 1001 1000 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1003 1002 1001 1000 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program