/var/tmp/bins/crutest verify -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 --verify-out /tmp/test_repair_verify.data --verify-in /tmp/test_repair_verify.data --range -q -g 199 {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.864957366Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"6b815a43cfc10bcc32090edef97cce2a57967dc5\",\n git_commit_timestamp: \"2024-07-10T15:11:32.000000000Z\",\n git_branch: \"renovate/nbd-0.x\",\n rustc_semver: \"1.76.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"07dca489ac2d933c78d3c5158e3f43beefeb02ce\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.86528457Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978} {"msg":"Upstairs <-> Downstairs Message Version: 8","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.86530438Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978} {"msg":"Crucible 8ecff375-2043-4164-b475-a05994458299 has session id: dbb1e7bb-ef81-48bf-9fee-467e53eb66f1","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.865652195Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Upstairs opts: Upstairs UUID: 8ecff375-2043-4164-b475-a05994458299, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.865699985Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Crucible stats registered with UUID: 8ecff375-2043-4164-b475-a05994458299","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.865719646Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} Crucible runtime is spawned {"msg":"The guest has requested activation with gen:199","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.86608951Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978} {"msg":"8ecff375-2043-4164-b475-a05994458299 active request set","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.86611161Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.866126531Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"0","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.866141221Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"0","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.866155021Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.866168731Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.866190751Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.866215812Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.866244332Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"io task","client":"0","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.866268952Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.866290903Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"io task","client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"ds_connection connected from Ok(127.0.0.1:64367)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.866548116Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"io task","client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"ds_connection connected from Ok(127.0.0.1:37829)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.866770859Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"io task","client":"0","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-07-10T16:06:27.866816119Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-07-10T16:06:27.86684332Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-07-10T16:06:27.86687016Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Gone missing, transition from New to Disconnected","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.866893131Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Disconnected -> New","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.866917881Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.867056473Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.867320886Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.867345366Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.867480888Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"0","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.867500328Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"0","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.867928314Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.868099916Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"0","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.868514071Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.868532881Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.868546772Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.868608122Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"0","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.868625822Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"0","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:27.868647463Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.86922046Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"ds_connection connected from Ok(127.0.0.1:53331)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.869340262Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.870251613Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.870278854Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.87079984Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871351187Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871375368Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[0]R flush_numbers[0..12]: [406, 406, 403, 403, 405, 405, 405, 397, 402, 402, 387, 404]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871400108Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[0]R generation[0..12]: [198, 198, 196, 196, 197, 197, 197, 194, 196, 196, 188, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871418548Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[0]R dirty[0..12]: [false, true, false, false, true, true, false, true, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871436438Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[1]R flush_numbers[0..12]: [406, 406, 403, 403, 405, 405, 405, 397, 402, 402, 387, 404]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871454109Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[1]R generation[0..12]: [198, 198, 196, 196, 197, 197, 197, 194, 196, 196, 188, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871471559Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[1]R dirty[0..12]: [false, true, false, false, true, true, false, true, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871489179Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[2]R flush_numbers[0..12]: [406, 406, 403, 403, 405, 405, 405, 397, 402, 402, 387, 404]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871506849Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[2]R generation[0..12]: [198, 198, 196, 196, 197, 197, 197, 194, 196, 196, 188, 196]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.87152419Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"[2]R dirty[0..12]: [false, true, false, false, true, true, false, true, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.87154181Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Max found gen is 199","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.87155799Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Generation requested: 199 >= found:199","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.87157464Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Next flush: 407","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.87159093Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Extents 1 dirty","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.87160726Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"First source client ID for extent 1","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871624631Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:1 gens: 198 198 198","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871645231Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:1 flush: 406 406 406 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871664571Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871683452Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:1 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871701522Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"find dest for source 0 for extent at index 1","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871719642Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871737642Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871755622Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Extents 4 dirty","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871774113Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"First source client ID for extent 4","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871791233Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:4 gens: 197 197 197","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871809153Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:4 flush: 405 405 405 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871827983Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871846434Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:4 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871864514Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"find dest for source 0 for extent at index 4","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871882784Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871905474Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871934525Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Extents 5 dirty","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871953735Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"First source client ID for extent 5","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871971105Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:5 gens: 197 197 197","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.871988895Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:5 flush: 405 405 405 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872007296Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872025946Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:5 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872044156Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"find dest for source 0 for extent at index 5","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872062406Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872080217Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872096917Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Extents 7 dirty","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872111047Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"First source client ID for extent 7","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872124957Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:7 gens: 194 194 194","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872139467Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:7 flush: 397 397 397 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872154518Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872171808Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:7 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872186398Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"find dest for source 0 for extent at index 7","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872200918Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872215378Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872229688Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Extents 12 dirty","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872244148Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"First source client ID for extent 12","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872257859Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:12 gens: 196 196 196","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872272209Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:12 flush: 402 402 402 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872287169Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872301779Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:12 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872316149Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"find dest for source 0 for extent at index 12","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.87233078Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.87234516Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.87235957Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Extents 23 dirty","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.87237572Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"First source client ID for extent 23","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872389701Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:23 gens: 198 198 198","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872404031Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:23 flush: 406 406 406 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872418871Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872433581Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"extent:23 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872447801Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"find dest for source 0 for extent at index 23","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872462301Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872476582Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872490922Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872505722Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"0","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872519192Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872532442Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"starting reconciliation 8cc8e902-cfd9-4c47-8bfc-0f4a93161524: found 6 extents that need repair","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872546763Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Full repair list: {ExtentId(5): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(4): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(12): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(23): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(1): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(7): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }}","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872567483Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Task list: [ReconcileIO { id: ReconciliationId(0), op: ExtentFlush { repair_id: ReconciliationId(0), extent_id: ExtentId(5), client_id: ClientId(0), flush_number: 407, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(1), op: ExtentClose { repair_id: ReconciliationId(1), extent_id: ExtentId(5) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(2), op: ExtentRepair { repair_id: ReconciliationId(2), extent_id: ExtentId(5), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(3), op: ExtentReopen { repair_id: ReconciliationId(3), extent_id: ExtentId(5) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(4), op: ExtentFlush { repair_id: ReconciliationId(4), extent_id: ExtentId(4), client_id: ClientId(0), flush_number: 407, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(5), op: ExtentClose { repair_id: ReconciliationId(5), extent_id: ExtentId(4) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(6), op: ExtentRepair { repair_id: ReconciliationId(6), extent_id: ExtentId(4), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(7), op: ExtentReopen { repair_id: ReconciliationId(7), extent_id: ExtentId(4) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(8), op: ExtentFlush { repair_id: ReconciliationId(8), extent_id: ExtentId(12), client_id: ClientId(0), flush_number: 407, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(9), op: ExtentClose { repair_id: ReconciliationId(9), extent_id: ExtentId(12) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(10), op: ExtentRepair { repair_id: ReconciliationId(10), extent_id: ExtentId(12), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(11), op: ExtentReopen { repair_id: ReconciliationId(11), extent_id: ExtentId(12) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(12), op: ExtentFlush { repair_id: ReconciliationId(12), extent_id: ExtentId(23), client_id: ClientId(0), flush_number: 407, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(13), op: ExtentClose { repair_id: ReconciliationId(13), extent_id: ExtentId(23) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(14), op: ExtentRepair { repair_id: ReconciliationId(14), extent_id: ExtentId(23), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(15), op: ExtentReopen { repair_id: ReconciliationId(15), extent_id: ExtentId(23) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(16), op: ExtentFlush { repair_id: ReconciliationId(16), extent_id: ExtentId(1), client_id: ClientId(0), flush_number: 407, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(17), op: ExtentClose { repair_id: ReconciliationId(17), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(18), op: ExtentRepair { repair_id: ReconciliationId(18), extent_id: ExtentId(1), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(19), op: ExtentReopen { repair_id: ReconciliationId(19), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(20), op: ExtentFlush { repair_id: ReconciliationId(20), extent_id: ExtentId(7), client_id: ClientId(0), flush_number: 407, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(21), op: ExtentClose { repair_id: ReconciliationId(21), extent_id: ExtentId(7) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(22), op: ExtentRepair { repair_id: ReconciliationId(22), extent_id: ExtentId(7), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(23), op: ExtentReopen { repair_id: ReconciliationId(23), extent_id: ExtentId(7) }, state: ClientData([New, New, New]) }]","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.872750125Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.873368203Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.873402453Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.886258117Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.886293357Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.899037369Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.899065749Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.910621196Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.910649096Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.92278293Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.922821661Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending reconcile request ReconciliationId(22)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.935444431Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"sending reconcile request ReconciliationId(22)","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.935497671Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"done with reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.945890893Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"":"downstairs","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.945924184Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"0","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} Wait for a query_work_queue command to finish before sending IO {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.945949054Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"1","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.945967064Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"client":"2","":"downstairs","session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.945984374Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.945999924Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"8ecff375-2043-4164-b475-a05994458299 is now active with session: dbb1e7bb-ef81-48bf-9fee-467e53eb66f1","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.946016415Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978,"session_id":"dbb1e7bb-ef81-48bf-9fee-467e53eb66f1"} {"msg":"The guest has finished waiting for activation with:199","v":0,"name":"crucible","level":30,"time":"2024-07-10T16:06:37.946032045Z","hostname":"w-01J2EKZDSC9YSAZ70AS3VCQEFS","pid":1978} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Read and Verify all blocks (0..600 range:true) Wrote out file "/tmp/test_repair_verify.data" Verify test completed Wrote out file "/tmp/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1 Acked 1000 Read 88 Done Done Done false 2 Acked 1001 Read 256 Done Done Done false 3 Acked 1002 Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL New 0 0 0 0 Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 0 0 0 Downstairs last five completed: Upstairs last five completed: 3 2 1 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 4 3 2 1 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program