/var/tmp/bins/crutest verify -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 --verify-out /tmp/test_repair_verify.data --verify-in /tmp/test_repair_verify.data --range -q -g 199 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:36.982364141Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2024-11-20T14:36:37.0334163Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.033891004Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.070310186Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.070954142Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"e10e1f0ceaa5049614af941033fd2935730f49b1\",\n git_commit_timestamp: \"2024-11-20T14:12:42.000000000Z\",\n git_branch: \"alan/fix-header\",\n rustc_semver: \"1.82.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"f6e511eec7342f59a25f7c0534f1dbea00d01b14\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.071001605Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987} {"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.071020422Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.071038929Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987} {"msg":"Crucible 10baebae-48b6-4a12-a450-b90206848d18 has session id: 3367091c-189c-44e6-a60d-8d8dd508666d","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.0718631Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Upstairs opts: Upstairs UUID: 10baebae-48b6-4a12-a450-b90206848d18, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.071910443Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Crucible stats registered with UUID: 10baebae-48b6-4a12-a450-b90206848d18","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.071931584Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.106803423Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"task":"crutest"} {"msg":"The guest has requested activation with gen:199","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.106866617Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987} {"msg":"10baebae-48b6-4a12-a450-b90206848d18 active request set","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.106998487Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.107097241Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"0","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.107187138Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"0","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.107248949Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.107306011Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.107623786Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.107693934Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.107742009Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"io task","client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.107790654Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"io task","client":"0","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.107841835Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"io task","client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"ds_connection connected from Ok(127.0.0.1:39580)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.107899479Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"io task","client":"0","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"ds_connection connected from Ok(127.0.0.1:58120)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.107955759Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"io task","client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"ds_connection connected from Ok(127.0.0.1:58612)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.107988764Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"io task","client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.108392889Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"0","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.108441535Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"0","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.108478277Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.108513706Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.108551531Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.108585057Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.109439948Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.109489275Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.109525807Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"0","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110030038Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.11007169Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110090557Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.11017825Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110201335Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110217587Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110488209Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"0","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110510523Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"0","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[0]R flush_numbers[0..12]: [326, 323, 328, 322, 320, 320, 320, 320, 327, 325, 319, 320]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110580391Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[0]R generation[0..12]: [198, 196, 198, 196, 194, 194, 194, 194, 198, 198, 193, 194]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110602725Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, true, false, false, true, false]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110626431Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[1]R flush_numbers[0..12]: [326, 323, 328, 322, 320, 320, 320, 320, 327, 325, 319, 320]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110644076Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[1]R generation[0..12]: [198, 196, 198, 196, 194, 194, 194, 194, 198, 198, 193, 194]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110660689Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, true, false, false, true, false]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110677762Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[2]R flush_numbers[0..12]: [326, 323, 328, 322, 320, 320, 320, 320, 327, 325, 319, 320]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110694195Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[2]R generation[0..12]: [198, 196, 198, 196, 194, 194, 194, 194, 198, 198, 193, 194]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110710917Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, true, false, false, true, false]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.1107277Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Max found gen is 199","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110743311Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Generation requested: 199 >= found:199","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110759794Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Next flush: 329","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110775705Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Extents 7 dirty","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110791516Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"First source client ID for extent 7","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.11080873Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:7 gens: 194 194 194","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110827096Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:7 flush: 320 320 320 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110844871Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.11086462Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:7 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110881894Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"find dest for source 0 for extent at index 7","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110898987Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110916231Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110932784Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Extents 10 dirty","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110963234Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"First source client ID for extent 10","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.110981379Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:10 gens: 193 193 193","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111021558Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:10 flush: 319 319 319 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111040987Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.11105816Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:10 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111074973Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"find dest for source 0 for extent at index 10","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111092137Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.11110869Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111125152Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Extents 17 dirty","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111141845Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"First source client ID for extent 17","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111161674Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:17 gens: 196 196 196","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111179098Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:17 flush: 323 323 323 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111196723Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111214007Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:17 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.1112309Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"find dest for source 0 for extent at index 17","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111248014Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111265238Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111282271Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Extents 19 dirty","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111299024Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"First source client ID for extent 19","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111315426Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:19 gens: 193 193 193","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111332239Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:19 flush: 319 319 319 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111349844Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111366948Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:19 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.11138366Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"find dest for source 0 for extent at index 19","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.11140392Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111421074Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111437877Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Extents 21 dirty","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.11145449Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"First source client ID for extent 21","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111470662Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:21 gens: 194 194 194","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111487435Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:21 flush: 320 320 320 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111505109Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111522393Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:21 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111539397Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"find dest for source 0 for extent at index 21","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.11155652Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111573604Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111590607Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Extents 22 dirty","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.11160737Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"First source client ID for extent 22","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111623672Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:22 gens: 178 178 178","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111640686Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:22 flush: 295 295 295 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111661547Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111679071Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:22 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111695884Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"find dest for source 0 for extent at index 22","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111712627Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.11172918Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111745903Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Extents 23 dirty","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111762996Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"First source client ID for extent 23","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111779068Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:23 gens: 194 194 194","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111796041Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:23 flush: 320 320 320 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111813235Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111830249Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"extent:23 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111846972Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"find dest for source 0 for extent at index 23","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111864085Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.111881019Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.11202904Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.112074299Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"0","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.112107294Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.112126922Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"starting reconciliation d8b013a4-975f-4e6b-9ece-1e467775d231: found 7 extents that need repair","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.112147302Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Full repair list: {ExtentId(21): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(17): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(23): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(22): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(19): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(7): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(10): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }}","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.112187Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Task list: [ReconcileIO { id: ReconciliationId(0), op: ExtentFlush { repair_id: ReconciliationId(0), extent_id: ExtentId(21), client_id: ClientId(0), flush_number: 329, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(1), op: ExtentClose { repair_id: ReconciliationId(1), extent_id: ExtentId(21) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(2), op: ExtentRepair { repair_id: ReconciliationId(2), extent_id: ExtentId(21), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(3), op: ExtentReopen { repair_id: ReconciliationId(3), extent_id: ExtentId(21) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(4), op: ExtentFlush { repair_id: ReconciliationId(4), extent_id: ExtentId(17), client_id: ClientId(0), flush_number: 329, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(5), op: ExtentClose { repair_id: ReconciliationId(5), extent_id: ExtentId(17) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(6), op: ExtentRepair { repair_id: ReconciliationId(6), extent_id: ExtentId(17), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(7), op: ExtentReopen { repair_id: ReconciliationId(7), extent_id: ExtentId(17) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(8), op: ExtentFlush { repair_id: ReconciliationId(8), extent_id: ExtentId(23), client_id: ClientId(0), flush_number: 329, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(9), op: ExtentClose { repair_id: ReconciliationId(9), extent_id: ExtentId(23) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(10), op: ExtentRepair { repair_id: ReconciliationId(10), extent_id: ExtentId(23), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(11), op: ExtentReopen { repair_id: ReconciliationId(11), extent_id: ExtentId(23) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(12), op: ExtentFlush { repair_id: ReconciliationId(12), extent_id: ExtentId(22), client_id: ClientId(0), flush_number: 329, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(13), op: ExtentClose { repair_id: ReconciliationId(13), extent_id: ExtentId(22) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(14), op: ExtentRepair { repair_id: ReconciliationId(14), extent_id: ExtentId(22), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(15), op: ExtentReopen { repair_id: ReconciliationId(15), extent_id: ExtentId(22) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(16), op: ExtentFlush { repair_id: ReconciliationId(16), extent_id: ExtentId(19), client_id: ClientId(0), flush_number: 329, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(17), op: ExtentClose { repair_id: ReconciliationId(17), extent_id: ExtentId(19) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(18), op: ExtentRepair { repair_id: ReconciliationId(18), extent_id: ExtentId(19), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(19), op: ExtentReopen { repair_id: ReconciliationId(19), extent_id: ExtentId(19) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(20), op: ExtentFlush { repair_id: ReconciliationId(20), extent_id: ExtentId(7), client_id: ClientId(0), flush_number: 329, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(21), op: ExtentClose { repair_id: ReconciliationId(21), extent_id: ExtentId(7) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(22), op: ExtentRepair { repair_id: ReconciliationId(22), extent_id: ExtentId(7), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(23), op: ExtentReopen { repair_id: ReconciliationId(23), extent_id: ExtentId(7) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(24), op: ExtentFlush { repair_id: ReconciliationId(24), extent_id: ExtentId(10), client_id: ClientId(0), flush_number: 329, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(25), op: ExtentClose { repair_id: ReconciliationId(25), extent_id: ExtentId(10) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(26), op: ExtentRepair { repair_id: ReconciliationId(26), extent_id: ExtentId(10), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(27), op: ExtentReopen { repair_id: ReconciliationId(27), extent_id: ExtentId(10) }, state: ClientData([New, New, New]) }]","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.112324972Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting start notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.112350291Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.11238524Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.112407033Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.112428996Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.112445829Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.112462151Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.119266102Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.119962301Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.120283403Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.120564606Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.1206026Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.120621377Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.125714313Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.126389771Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.126759819Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.127067013Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.127091962Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.127109357Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.132279314Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.132781782Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.133132892Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.133461248Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.133493301Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.133512749Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.138525548Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.13907612Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.139440537Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.139750487Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.139783642Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.139871424Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.144850626Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.145283237Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.145692091Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(22)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.146041248Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(22)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.146079183Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.14609813Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.158124431Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.158710493Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.159109498Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(26)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.159492972Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"sending reconcile request ReconciliationId(26)","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.159529123Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.159611665Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.164722687Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"done with reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.165091552Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.165121451Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"0","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.165140328Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"1","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.165157311Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"client":"2","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.165174345Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.165189986Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"10baebae-48b6-4a12-a450-b90206848d18 is now active with session: 3367091c-189c-44e6-a60d-8d8dd508666d","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.165206168Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} {"msg":"The guest has finished waiting for activation with:199","v":0,"name":"crucible","level":30,"time":"2024-11-20T14:36:37.165221558Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987} {"msg":"no Nexus client from DNS, aborting finish notification","v":0,"name":"crucible","level":50,"time":"2024-11-20T14:36:37.165251968Z","hostname":"bmat-EVT22200007-00004b2e","pid":1987,"reconcile":"d8b013a4-975f-4e6b-9ece-1e467775d231","":"downstairs","":"downstairs","session_id":"3367091c-189c-44e6-a60d-8d8dd508666d"} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Read and Verify all blocks (0..600 range:true) Wrote out file "/tmp/test_repair_verify.data" Verify test completed Wrote out file "/tmp/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Read 88 Done Done Done false 1001 Acked Read 256 Done Done Done false 1002 Acked Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 0 0 0 Downstairs last five completed: Upstairs last five completed: 1002 1001 1000 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1003 1002 1001 1000 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program