/var/tmp/bins/crutest verify -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 --verify-out /tmp/test_repair_verify.data --verify-in /tmp/test_repair_verify.data --range -q -g 199 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.812827891Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.856579194Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.857279126Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"26e7d501b33f47379e53601720353c2983e7654b\",\n git_commit_timestamp: \"2024-12-12T20:16:40.000000000Z\",\n git_branch: \"mkeeter/lean-mean-state-machine\",\n rustc_semver: \"1.82.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"f6e511eec7342f59a25f7c0534f1dbea00d01b14\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.857314083Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978} {"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.857328062Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.857342041Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978} {"msg":"Crucible aecf9e9e-39fa-4eb6-8fcb-e657a0707c33 has session id: 7019f6ab-9b3d-4496-be86-1f007f839d5e","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.857728975Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Upstairs opts: Upstairs UUID: aecf9e9e-39fa-4eb6-8fcb-e657a0707c33, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.857774861Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Crucible stats registered with UUID: aecf9e9e-39fa-4eb6-8fcb-e657a0707c33","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.85779337Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888337324Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"task":"crutest"} {"msg":"The guest has requested activation with gen:199","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888363472Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978} {"msg":"aecf9e9e-39fa-4eb6-8fcb-e657a0707c33 active request set","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888424388Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888441477Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"0","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888465395Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"0","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888483634Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888498113Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888511272Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting...","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888525101Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.88853818Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"io task","client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888551759Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"io task","client":"0","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888565228Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"ds_connection connected from Ok(127.0.0.1:44853)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888746866Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"io task","client":"0","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"ds_connection connected from Ok(127.0.0.1:61367)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888788243Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"ds_connection connected from Ok(127.0.0.1:44438)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.888882696Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"io task","client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.889513573Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.889541751Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"0","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"version negotiation from state Connecting { state: WaitForPromote, mode: New }","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.88955679Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.890177537Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"0","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.890209115Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.890224654Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.890721769Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.89085975Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"[0]R flush_numbers[0..12]: [361, 373, 373, 370, 370, 374, 374, 375, 367, 367, 362, 354]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.890900627Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"[0]R generation[0..12]: [192, 198, 198, 196, 196, 198, 198, 198, 195, 195, 193, 189]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.890917536Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.890932545Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"[1]R flush_numbers[0..12]: [361, 373, 373, 370, 370, 374, 374, 375, 367, 367, 362, 354]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.890947844Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"[1]R generation[0..12]: [192, 198, 198, 196, 196, 198, 198, 198, 195, 195, 193, 189]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.890962033Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.890976022Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"[2]R flush_numbers[0..12]: [361, 373, 373, 370, 370, 374, 374, 375, 367, 367, 362, 354]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.89100362Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"[2]R generation[0..12]: [192, 198, 198, 196, 196, 198, 198, 198, 195, 195, 193, 189]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891019739Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891033968Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Max found gen is 199","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891046937Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Generation requested: 199 >= found:199","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891064366Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Next flush: 376","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891077735Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Extents 17 dirty","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891090744Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"First source client ID for extent 17","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891104363Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:17 gens: 197 197 197","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891118782Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:17 flush: 372 372 372 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891133531Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.89114825Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:17 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891162559Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"find dest for source 0 for extent at index 17","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891176948Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891191047Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891205056Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Extents 18 dirty","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891227155Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"First source client ID for extent 18","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891241724Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:18 gens: 197 197 197","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891255973Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:18 flush: 372 372 372 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891270542Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891288221Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:18 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891302849Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"find dest for source 0 for extent at index 18","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891317208Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891331287Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891345546Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Extents 19 dirty","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891359616Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"First source client ID for extent 19","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891373135Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:19 gens: 197 197 197","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891387254Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:19 flush: 372 372 372 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891401782Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891416152Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:19 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891430301Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"find dest for source 0 for extent at index 19","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.89144452Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891458549Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891472698Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Extents 20 dirty","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891488527Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"First source client ID for extent 20","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891502316Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:20 gens: 197 197 197","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891516375Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:20 flush: 372 372 372 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891530904Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891545273Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:20 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891559342Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"find dest for source 0 for extent at index 20","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891573471Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.89158734Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891601369Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Extents 27 dirty","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891615298Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"First source client ID for extent 27","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891628877Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:27 gens: 195 195 195","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891649245Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:27 flush: 367 367 367 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891678134Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891695442Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:27 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891712371Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"find dest for source 0 for extent at index 27","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.89172764Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891741649Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891755798Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Extents 28 dirty","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891770037Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"First source client ID for extent 28","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891783566Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:28 gens: 196 196 196","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891797495Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:28 flush: 370 370 370 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891812044Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891826593Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"extent:28 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891840532Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"find dest for source 0 for extent at index 28","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891854951Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.89186883Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891882909Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891897748Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"0","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891911407Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Transition from Connecting { state: WaitQuorum, mode: New } to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891927716Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Full repair list: {ExtentId(17): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(27): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(28): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(18): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(19): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(20): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }}","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.891947735Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Task list: [ReconcileIO { id: ReconciliationId(0), op: ExtentFlush { repair_id: ReconciliationId(0), extent_id: ExtentId(17), client_id: ClientId(0), flush_number: 376, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(1), op: ExtentClose { repair_id: ReconciliationId(1), extent_id: ExtentId(17) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(2), op: ExtentRepair { repair_id: ReconciliationId(2), extent_id: ExtentId(17), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(3), op: ExtentReopen { repair_id: ReconciliationId(3), extent_id: ExtentId(17) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(4), op: ExtentFlush { repair_id: ReconciliationId(4), extent_id: ExtentId(27), client_id: ClientId(0), flush_number: 376, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(5), op: ExtentClose { repair_id: ReconciliationId(5), extent_id: ExtentId(27) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(6), op: ExtentRepair { repair_id: ReconciliationId(6), extent_id: ExtentId(27), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(7), op: ExtentReopen { repair_id: ReconciliationId(7), extent_id: ExtentId(27) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(8), op: ExtentFlush { repair_id: ReconciliationId(8), extent_id: ExtentId(28), client_id: ClientId(0), flush_number: 376, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(9), op: ExtentClose { repair_id: ReconciliationId(9), extent_id: ExtentId(28) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(10), op: ExtentRepair { repair_id: ReconciliationId(10), extent_id: ExtentId(28), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(11), op: ExtentReopen { repair_id: ReconciliationId(11), extent_id: ExtentId(28) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(12), op: ExtentFlush { repair_id: ReconciliationId(12), extent_id: ExtentId(18), client_id: ClientId(0), flush_number: 376, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(13), op: ExtentClose { repair_id: ReconciliationId(13), extent_id: ExtentId(18) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(14), op: ExtentRepair { repair_id: ReconciliationId(14), extent_id: ExtentId(18), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(15), op: ExtentReopen { repair_id: ReconciliationId(15), extent_id: ExtentId(18) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(16), op: ExtentFlush { repair_id: ReconciliationId(16), extent_id: ExtentId(19), client_id: ClientId(0), flush_number: 376, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(17), op: ExtentClose { repair_id: ReconciliationId(17), extent_id: ExtentId(19) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(18), op: ExtentRepair { repair_id: ReconciliationId(18), extent_id: ExtentId(19), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(19), op: ExtentReopen { repair_id: ReconciliationId(19), extent_id: ExtentId(19) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(20), op: ExtentFlush { repair_id: ReconciliationId(20), extent_id: ExtentId(20), client_id: ClientId(0), flush_number: 376, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(21), op: ExtentClose { repair_id: ReconciliationId(21), extent_id: ExtentId(20) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(22), op: ExtentRepair { repair_id: ReconciliationId(22), extent_id: ExtentId(20), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(23), op: ExtentReopen { repair_id: ReconciliationId(23), extent_id: ExtentId(20) }, state: ClientData([New, New, New]) }]","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.892043318Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"starting reconciliation a4221d49-ee36-48e4-aede-ae25c95c2a35: found 24 extents that need repair","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.892061027Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.892074986Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting start notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.892091085Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.892105674Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.892120103Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.892133372Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.892146601Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.896568986Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.897008206Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.897268648Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.897536279Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.897574277Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.897604715Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.90130068Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.901676464Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.901889719Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.90217909Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.902203198Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.902218037Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.905729954Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.906076531Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.906342392Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.906643471Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.906673289Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.906689668Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.909973612Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.910404412Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.910686733Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.911013Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending reconcile request ReconciliationId(18)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.911032279Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.911048488Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.916583076Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.916951661Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.917228072Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending reconcile request ReconciliationId(22)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.917519712Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"sending reconcile request ReconciliationId(22)","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.917560419Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.917592536Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.921037389Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"done with reconciliation","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.921332239Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.921361347Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"0","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.921376736Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"1","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.921390375Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"client":"2","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.921403974Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.921416383Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"aecf9e9e-39fa-4eb6-8fcb-e657a0707c33 is now active with session: 7019f6ab-9b3d-4496-be86-1f007f839d5e","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.921429162Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} {"msg":"The guest has finished waiting for activation with:199","v":0,"name":"crucible","level":30,"time":"2024-12-12T20:40:26.921441531Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978} {"msg":"no Nexus client from DNS, aborting finish notification","v":0,"name":"crucible","level":50,"time":"2024-12-12T20:40:26.92145274Z","hostname":"w-01JEY94CKQJCK3GEXVRRGZA3GF","pid":1978,"reconcile":"a4221d49-ee36-48e4-aede-ae25c95c2a35","":"downstairs","":"downstairs","session_id":"7019f6ab-9b3d-4496-be86-1f007f839d5e"} Wait for a query_work_queue command to finish before sending IO Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Read and Verify all blocks (0..600 range:true) Wrote out file "/tmp/test_repair_verify.data" Verify test completed Wrote out file "/tmp/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Read 88 Done Done Done false 1001 Acked Read 256 Done Done Done false 1002 Acked Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 0 0 0 Downstairs last five completed: Upstairs last five completed: 1002 1001 1000 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1003 1002 1001 1000 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program