/var/tmp/bins/crutest verify -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 --verify-out /tmp/test_repair_verify.data --verify-in /tmp/test_repair_verify.data --range -q -g 199 look at: http://127.0.0.1:12810 RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 } {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:23.967524582Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"9d0029dbf9065191a369b89880e9a5f8aa8ee534\",\n git_commit_timestamp: \"2024-09-26T13:46:09.000000000Z\",\n git_branch: \"mkeeter/remove-iostatecount-new\",\n rustc_semver: \"1.80.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"051478957371ee0084a7c0913941d2a8c4757bb9\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:23.967858153Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978} {"msg":"Upstairs <-> Downstairs Message Version: 11","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:23.967878022Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:23.967893361Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978} {"msg":"Crucible 3eb8dd91-8363-4a79-8f60-09d9e24eda3c has session id: f7557379-c206-4ad2-8361-b257eabebafa","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:23.968109439Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Upstairs opts: Upstairs UUID: 3eb8dd91-8363-4a79-8f60-09d9e24eda3c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:23.968158676Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Crucible stats registered with UUID: 3eb8dd91-8363-4a79-8f60-09d9e24eda3c","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:23.968184105Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"The guest has requested activation with gen:199","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:23.999792045Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978} {"msg":"3eb8dd91-8363-4a79-8f60-09d9e24eda3c active request set","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:23.999833662Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:23.999865721Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"0","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:23.999911878Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"0","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:23.999947146Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:23.999979284Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.000382071Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.000428399Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.000452047Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"io task","client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.000467136Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"io task","client":"0","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.000481796Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"ds_connection connected from Ok(127.0.0.1:49805)","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.000505114Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"io task","client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"ds_connection connected from Ok(127.0.0.1:49024)","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.000533373Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"io task","client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"ds_connection connected from Ok(127.0.0.1:59547)","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.000560531Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"io task","client":"0","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.001219234Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.001247862Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.001263591Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"0","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.001303119Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"0","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.001329538Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.001355806Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.001874767Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002051767Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002086505Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"0","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002377828Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002406547Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002422086Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002584907Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"0","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002623644Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"0","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002651513Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.00271034Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002742958Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[0]R flush_numbers[0..12]: [322, 322, 334, 334, 329, 336, 336, 336, 331, 334, 328, 328]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002759047Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[0]R generation[0..12]: [189, 189, 198, 198, 194, 198, 198, 198, 195, 198, 193, 193]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002773936Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[0]R dirty[0..12]: [false, true, false, false, false, false, false, false, false, false, true, false]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002788765Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[1]R flush_numbers[0..12]: [322, 322, 334, 334, 329, 336, 336, 336, 331, 334, 328, 328]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002803264Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[1]R generation[0..12]: [189, 189, 198, 198, 194, 198, 198, 198, 195, 198, 193, 193]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002817353Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[1]R dirty[0..12]: [false, true, false, false, false, false, false, false, false, false, true, false]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002835712Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[2]R flush_numbers[0..12]: [322, 322, 334, 334, 329, 336, 336, 336, 331, 334, 328, 328]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002849881Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[2]R generation[0..12]: [189, 189, 198, 198, 194, 198, 198, 198, 195, 198, 193, 193]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002863731Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"[2]R dirty[0..12]: [false, true, false, false, false, false, false, false, false, false, true, false]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.00287791Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Max found gen is 199","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002891139Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Generation requested: 199 >= found:199","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002904389Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Next flush: 337","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002917388Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Extents 1 dirty","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002930427Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"First source client ID for extent 1","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002944126Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"extent:1 gens: 189 189 189","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002958235Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"extent:1 flush: 322 322 322 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002972945Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.002987364Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"extent:1 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003001643Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"find dest for source 0 for extent at index 1","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003015982Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003032151Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003046261Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Extents 10 dirty","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003070239Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"First source client ID for extent 10","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003094708Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"extent:10 gens: 193 193 193","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003123156Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"extent:10 flush: 328 328 328 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003152684Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003182353Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"extent:10 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003207211Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"find dest for source 0 for extent at index 10","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.00323306Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003257529Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003272698Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Extents 19 dirty","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003296096Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"First source client ID for extent 19","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003320055Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"extent:19 gens: 192 192 192","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003344024Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"extent:19 flush: 327 327 327 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003360253Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003377842Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"extent:19 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003392191Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"find dest for source 0 for extent at index 19","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.00340668Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003420839Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003434888Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Extents 20 dirty","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003449258Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"First source client ID for extent 20","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003463057Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"extent:20 gens: 191 191 191","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003477076Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"extent:20 flush: 324 324 324 scs: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003491785Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"max_flush now has: [ClientId(0), ClientId(1), ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003506054Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"extent:20 dirty: true true true","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003520194Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"find dest for source 0 for extent at index 20","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003534563Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"source 0, add dest 1 source flush","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003548862Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"source 0, add dest 2 source flush","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003562911Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"mrl":"dirty","":"mend","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.00357936Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"0","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003592879Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Transition from WaitQuorum to Reconcile","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003606119Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"starting reconciliation 56b6cce8-c5b0-4224-9b12-d171d8d3bd28: found 4 extents that need repair","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003620068Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Full repair list: {ExtentId(19): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(20): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(10): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }, ExtentId(1): ExtentFix { source: ClientId(0), dest: [ClientId(1), ClientId(2)] }}","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003639127Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Task list: [ReconcileIO { id: ReconciliationId(0), op: ExtentFlush { repair_id: ReconciliationId(0), extent_id: ExtentId(19), client_id: ClientId(0), flush_number: 337, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(1), op: ExtentClose { repair_id: ReconciliationId(1), extent_id: ExtentId(19) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(2), op: ExtentRepair { repair_id: ReconciliationId(2), extent_id: ExtentId(19), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(3), op: ExtentReopen { repair_id: ReconciliationId(3), extent_id: ExtentId(19) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(4), op: ExtentFlush { repair_id: ReconciliationId(4), extent_id: ExtentId(20), client_id: ClientId(0), flush_number: 337, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(5), op: ExtentClose { repair_id: ReconciliationId(5), extent_id: ExtentId(20) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(6), op: ExtentRepair { repair_id: ReconciliationId(6), extent_id: ExtentId(20), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(7), op: ExtentReopen { repair_id: ReconciliationId(7), extent_id: ExtentId(20) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(8), op: ExtentFlush { repair_id: ReconciliationId(8), extent_id: ExtentId(10), client_id: ClientId(0), flush_number: 337, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(9), op: ExtentClose { repair_id: ReconciliationId(9), extent_id: ExtentId(10) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(10), op: ExtentRepair { repair_id: ReconciliationId(10), extent_id: ExtentId(10), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(11), op: ExtentReopen { repair_id: ReconciliationId(11), extent_id: ExtentId(10) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(12), op: ExtentFlush { repair_id: ReconciliationId(12), extent_id: ExtentId(1), client_id: ClientId(0), flush_number: 337, gen_number: 199 }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(13), op: ExtentClose { repair_id: ReconciliationId(13), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(14), op: ExtentRepair { repair_id: ReconciliationId(14), extent_id: ExtentId(1), source_client_id: ClientId(0), source_repair_address: 0.0.0.0:12810, dest_clients: [ClientId(1), ClientId(2)] }, state: ClientData([New, New, New]) }, ReconcileIO { id: ReconciliationId(15), op: ExtentReopen { repair_id: ReconciliationId(15), extent_id: ExtentId(1) }, state: ClientData([New, New, New]) }]","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.003705043Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting start notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.003721742Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.003737311Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.004772253Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.005039617Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"sending reconcile request ReconciliationId(2)","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.005079955Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.005109473Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.017436075Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.017868851Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.019242563Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.019517988Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"sending reconcile request ReconciliationId(6)","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.019549186Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.019566615Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.029737589Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.030204512Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.031700448Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.031991201Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"sending reconcile request ReconciliationId(10)","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.032040758Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.032069427Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.042201893Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.042587981Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.044039409Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.044289925Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"sending reconcile request ReconciliationId(14)","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.044319643Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.044340512Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"no Nexus client from DNS, aborting progress notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.054066241Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"done with reconciliation","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.054360114Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.054386113Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"0","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.054410801Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"1","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Transition from Reconcile to Active","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.05443723Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"client":"2","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.054459369Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.054477568Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"3eb8dd91-8363-4a79-8f60-09d9e24eda3c is now active with session: f7557379-c206-4ad2-8361-b257eabebafa","v":0,"name":"crucibleWait for a query_work_queue command to finish before sending IO ","level":30,"time":"2024-09-26T14:35:24.054503196Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} {"msg":"The guest has finished waiting for activation with:199","v":0,"name":"crucible","level":30,"time":"2024-09-26T14:35:24.054527265Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978} {"msg":"no Nexus client from DNS, aborting finish notification","v":0,"name":"crucible","level":50,"time":"2024-09-26T14:35:24.054548324Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978,"reconcile":"56b6cce8-c5b0-4224-9b12-d171d8d3bd28","":"downstairs","":"downstairs","session_id":"f7557379-c206-4ad2-8361-b257eabebafa"} Region: es:20 ec:30 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_repair_verify.data" Read and Verify all blocks (0..600 range:true) Wrote out file "/tmp/test_repair_verify.data" Verify test completed Wrote out file "/tmp/test_repair_verify.data" CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:3 States: Active Active Active GW_ID ACK DSID TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1 Acked 1000 Read 88 Done Done Done false 2 Acked 1001 Read 256 Done Done Done false 3 Acked 1002 Read 256 Done Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 0 0 0 Downstairs last five completed: Upstairs last five completed: 3 2 1 CLIENT: Up:0 ds:3 act:3 ---------------------------------------------------------------- Crucible gen:199 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 4 3 2 1 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-09-26T14:35:28.109111326Z","hostname":"w-01J8QBMA0P4PS841BFGZXS21Y3","pid":1978}