/var/tmp/bins/crutest verify -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 --verify-out /tmp/test_repair_verify.data --verify-in /tmp/test_repair_verify.data --range -q -g 199 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.726345533Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2024-12-04T13:37:52.769542166Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.769914727Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.802338914Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.802966875Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"0e1f7f60aff9f26b0d737c97120445edbded1ad3\",\n git_commit_timestamp: \"2024-12-04T13:14:36.000000000Z\",\n git_branch: \"renovate/tokio-util-0.x-lockfile\",\n rustc_semver: \"1.82.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"f6e511eec7342f59a25f7c0534f1dbea00d01b14\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.803005992Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978} {"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.803021561Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.80303659Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978} {"msg":"Crucible 12a4697e-bae6-4b99-a28e-b6d14815e263 has session id: ad6b61ce-54e3-423d-ba9b-955d9bda6c01","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.803651182Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Upstairs opts: Upstairs UUID: 12a4697e-bae6-4b99-a28e-b6d14815e263, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.80368585Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Crucible stats registered with UUID: 12a4697e-bae6-4b99-a28e-b6d14815e263","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.803702788Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.834639251Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"task":"crutest"} {"msg":"The guest has requested activation with gen:199","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.834672738Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978} {"msg":"12a4697e-bae6-4b99-a28e-b6d14815e263 active request set","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.834744533Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.834772301Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"0","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.834800739Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"0","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.834896011Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.834935508Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835182079Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835258333Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835285991Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"io task","client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835305349Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"io task","client":"0","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835320518Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"ds_connection connected from Ok(127.0.0.1:50469)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835336007Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"io task","client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"ds_connection connected from Ok(127.0.0.1:47661)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835350346Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"ds_connection connected from Ok(127.0.0.1:33000)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835367884Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"io task","client":"0","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835779123Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835807541Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835885534Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"0","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835924471Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"0","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835950819Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.835965288Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.836439281Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.836468919Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.836484748Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"0","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837041825Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837078282Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837094971Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.83710821Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837122188Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837135698Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837148046Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"0","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837161425Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"0","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[0]R flush_numbers[0..12]: [335, 336, 335, 309, 329, 337, 336, 336, 336, 326, 336, 336]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837175395Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[0]R generation[0..12]: [197, 198, 197, 183, 194, 198, 198, 198, 198, 193, 198, 198]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837189703Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[0]R dirty[0..12]: [true, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837207472Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[1]R flush_numbers[0..12]: [335, 336, 335, 309, 329, 337, 336, 336, 336, 326, 336, 336]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837222231Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[1]R generation[0..12]: [197, 198, 197, 183, 194, 198, 198, 198, 198, 193, 198, 198]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.83723646Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[1]R dirty[0..12]: [true, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837250758Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[2]R flush_numbers[0..12]: [335, 336, 335, 309, 329, 337, 336, 336, 336, 326, 336, 336]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837264708Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[2]R generation[0..12]: [197, 198, 197, 183, 194, 198, 198, 198, 198, 193, 198, 198]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837278496Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"[2]R dirty[0..12]: [true, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837292445Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Max found gen is 199","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837305764Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Generation requested: 199 >= found:199","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837319333Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Next flush: 338","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837332552Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Extents 0 dirty","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837345631Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"First source client ID for extent 0","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.83735962Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"extent:0 gens: 197 197 197","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837374519Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"extent:0 flush: 335 335 335 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837390058Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837434584Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"extent:0 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837464622Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"find dest for source 0 for extent at index 0","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.83749168Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837521048Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837539026Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Extents 27 dirty","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837559955Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"First source client ID for extent 27","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837574653Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"extent:27 gens: 194 194 194","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837589152Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"extent:27 flush: 330 330 330 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837604141Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.83761912Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"extent:27 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837633659Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"find dest for source 0 for extent at index 27","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837648268Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837662767Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837677325Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Extents 28 dirty","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837694304Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"First source client ID for extent 28","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837708713Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"extent:28 gens: 198 198 198","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837723042Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"extent:28 flush: 337 337 337 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837737791Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.83775251Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"extent:28 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837767089Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"find dest for source 0 for extent at index 28","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837781757Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837796136Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837814275Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837842763Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"0","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837872361Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837904018Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Full repair list: {ExtentId(27): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(28): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(0): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }}","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837930106Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Task list: [ReconcileIO { id: ReconciliationId(0), op: ExtentFlush { repair_id: ReconciliationId(0), extent_id: ExtentId(27), client_id: ClientId(0), flush_number: 338, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(1), op: ExtentClose { repair_id: ReconciliationId(1), extent_id: ExtentId(27) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(2), op: ExtentRepair { repair_id: ReconciliationId(2), extent_id: ExtentId(27), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(3), op: ExtentReopen { repair_id: ReconciliationId(3), extent_id: ExtentId(27) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(4), op: ExtentFlush { repair_id: ReconciliationId(4), extent_id: ExtentId(28), client_id: ClientId(0), flush_number: 338, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(5), op: ExtentClose { repair_id: ReconciliationId(5), extent_id: ExtentId(28) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(6), op: ExtentRepair { repair_id: ReconciliationId(6), extent_id: ExtentId(28), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(7), op: ExtentReopen { repair_id: ReconciliationId(7), extent_id: ExtentId(28) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(8), op: ExtentFlush { repair_id: ReconciliationId(8), extent_id: ExtentId(0), client_id: ClientId(0), flush_number: 338, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(9), op: ExtentClose { repair_id: ReconciliationId(9), extent_id: ExtentId(0) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(10), op: ExtentRepair { repair_id: ReconciliationId(10), extent_id: ExtentId(0), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(11), op: ExtentReopen { repair_id: ReconciliationId(11), extent_id: ExtentId(0) }, state: ClientData([New, New, New]) }]","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837980772Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"starting reconciliation d7796415-a5a0-4de9-a6eb-57edfb80c3ac: found 12 extents that need repair","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.837997551Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"no Nexus client from DNS, aborting start notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.838023979Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.838045587Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.838061346Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.838231053Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.838273219Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.838311296Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.84290738Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.84329483Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.84355208Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.843872605Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.843913942Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.843984107Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.847744125Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.848121716Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.848380166Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.848706371Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.848747548Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.848779645Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.852303582Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"done with reconciliation","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.852643206Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.852666184Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"0","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.852681033Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"1","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.852694752Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"client":"2","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.852708511Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.85272124Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"12a4697e-bae6-4b99-a28e-b6d14815e263 is now active with session: ad6b61ce-54e3-423d-ba9b-955d9bda6c01","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.852734499Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} {"msg":"The guest has finished waiting for activation with:199","v":0,"name":"crucible","level":30,"time":"2024-12-04T13:37:52.852748128Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978} Wait for a query_work_queue command to finish before sending IO {"msg":"no Nexus client from DNS, aborting finish notification","v":0,"name":"crucible","level":50,"time":"2024-12-04T13:37:52.852810783Z","hostname":"w-01JE8XRV43WMH3PK20W2S569M0","pid":1978,"reconcile":"d7796415-a5a0-4de9-a6eb-57edfb80c3ac","":"downstairs","":"downstairs","session_id":"ad6b61ce-54e3-423d-ba9b-955d9bda6c01"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Read and Verify all blocks (0..600 range:true) Wrote out file "/tmp/test_repair_verify.data" Verify test completed Wrote out file "/tmp/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Read 88 Done Done Done false 1001 Acked Read 256 Done Done Done false 1002 Acked Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 0 0 0 Downstairs last five completed: Upstairs last five completed: 1002 1001 1000 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1003 1002 1001 1000 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program