Turn off color for downstairs dump Removing existing region "/var/tmp/test_repair" Creating region directory at: /var/tmp/test_repair Creating dsc directory at: /tmp/test_live_repair/test_repair_dsc Downstairs region 0 created at /var/tmp/test_repair/8810 in 0.20505115 Downstairs region 1 created at /var/tmp/test_repair/8820 in 0.013460539 Downstairs region 2 created at /var/tmp/test_repair/8830 in 0.012324421 Created 3 regions /var/tmp/bins/crutest with -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 1134 1135 1136 Dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.730693267Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.747261024Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.749696578Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.749969887Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.749979275Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.749990893Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139} {"msg":"Crucible 850a2010-a236-4333-a9f8-c48197bbdd7b has session id: bc68e7a8-0103-4f6d-84e0-023ca61cab99","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.757116108Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"Upstairs opts: Upstairs UUID: 850a2010-a236-4333-a9f8-c48197bbdd7b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.757476091Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"Crucible stats registered with UUID: 850a2010-a236-4333-a9f8-c48197bbdd7b","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.757487229Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:17.75758587Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758095345Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"2","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758340739Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758351717Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758651631Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758702171Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139} {"msg":"850a2010-a236-4333-a9f8-c48197bbdd7b active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.75871095Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758719688Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"client":"0","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758729366Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"client":"1","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758738904Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"client":"2","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758748563Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"2","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758758471Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758768369Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"ds_connection connected from Ok(127.0.0.1:46228)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758861761Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"2","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"ds_connection connected from Ok(127.0.0.1:42994)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758872499Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"ds_connection connected from Ok(127.0.0.1:59562)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.758882727Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.760747188Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"client":"1","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.760768944Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"client":"2","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.760781682Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"client":"0","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761546499Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761578593Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761709198Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"downstairs","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761720426Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"downstairs","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761731374Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"downstairs","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761741522Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"downstairs","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.76175155Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"downstairs","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761762448Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"downstairs","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761772556Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"downstairs","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761782604Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"downstairs","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761793402Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"downstairs","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761802561Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"downstairs","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761812099Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"downstairs","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761821327Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"downstairs","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761832545Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"downstairs","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761842113Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761850822Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"client":"0","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.76186035Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"client":"1","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761869788Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"client":"2","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761879906Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761888884Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"850a2010-a236-4333-a9f8-c48197bbdd7b is now active with session: bc68e7a8-0103-4f6d-84e0-023ca61cab99","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761898573Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761907091Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.761915979Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139} {"msg":"Activated sub_volume 850a2010-a236-4333-a9f8-c48197bbdd7b","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.762363685Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.762406947Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 {"msg":"Fill test","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.762870491Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"task":"crutest"} Read and Verify all blocks (0..600 range:false) {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30---------------------------------------------------------------- ,"time":"2026-07-20T22:56:17.802685654Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"task":"crutest"} Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:3 {States:"msg":" ActiveCLIENT: Tests done. All submitted work has been ACK'd" Active," Activev ":0JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY ,"name":"crucible","level":30 1004 Acked Read 88," Donetime Done" Done:" false 2026-07-20T22:56:17.802733525Z"," 1005 Acked Read 256hostname Done" Done Done:" false w-01KY0VTEM5JBMR6FFPKK5ET1FH" 1006 Acked Read 256, Done" Done Donepid false ":1139,"task":"crutest"} STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 0 0 Done 3 3 3 9 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1003)) Some(JobId(1003)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1006 1005 1004 1003 1002 {"msg":"CLIENT: Up:0 ds:3 act:3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:17.802791974Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"task":"crutest"} ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 1007 1006 1005 1004 1003 Upstairs last five completed: 1007 1006 1005 1004 1003 {"msg":"CLIENT: Up:0 ds:0 act:3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.804574962Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"task":"crutest"} {"msg":"CLIENT: All crucible jobs finished, exiting program","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.804614045Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"task":"crutest"} {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:21.804623583Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139} {"msg":"client request queue closed unexpectedly; is the program exiting?","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:21.804673214Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"client stopping due to Err(RecvError(()))","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.804685821Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"client task is sending Done(QueueClosed)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:21.804697129Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"client task is sending Done(QueueClosed)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:21.804707927Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"client task could not reply to main task; shutting down?","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:21.804719335Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"client task could not reply to main task; shutting down?","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:21.804730293Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.804741101Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"0","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.804751729Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1139,"":"io task","client":"1","":"downstairs","session_id":"bc68e7a8-0103-4f6d-84e0-023ca61cab99"} Begin loop 1 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.828568058Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:21.835654851Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.835697103Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.842629835Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.842896295Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.842925179Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.842933668Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.842945006Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141} {"msg":"Crucible ecfa5332-9f62-4b49-b962-eb39fb148f16 has session id: cfe64458-c95d-41a9-b7f8-dd67d9fee52c","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843289441Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"Upstairs opts: Upstairs UUID: ecfa5332-9f62-4b49-b962-eb39fb148f16, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843310647Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"Crucible stats registered with UUID: ecfa5332-9f62-4b49-b962-eb39fb148f16","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843320865Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:21.843329794Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843471187Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"io task","client":"0","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843487404Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"io task","client":"1","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843497502Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"io task","client":"2","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843640785Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"task":"crutest"} {"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843660022Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141} {"msg":"ecfa5332-9f62-4b49-b962-eb39fb148f16 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843686297Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843695725Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"client":"0","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843705343Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"client":"1","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843714771Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"client":"2","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.84372413Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"io task","client":"2","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843752864Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"io task","client":"0","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.84377625Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"io task","client":"1","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"ds_connection connected from Ok(127.0.0.1:63956)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843790747Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"io task","client":"2","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"ds_connection connected from Ok(127.0.0.1:64423)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.843871392Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"io task","client":"1","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"ds_connection connected from Ok(127.0.0.1:42151)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.84388315Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"io task","client":"0","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844460742Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"client":"1","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844486167Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"client":"2","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844502784Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"client":"0","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.84478908Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844835652Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844881163Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"downstairs","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844892071Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"downstairs","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844902909Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"downstairs","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844912937Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"downstairs","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844922935Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"downstairs","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844933753Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"downstairs","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844943801Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"downstairs","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844953869Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"downstairs","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844964607Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"downstairs","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844973746Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"downstairs","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844983314Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"downstairs","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"Next flush: 2","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.844992552Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"downstairs","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.84500178Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"":"downstairs","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.845011539Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.845020107Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"client":"0","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.845031285Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"client":"1","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.845040883Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"client":"2","":"downstairs","session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.845050531Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.84505947Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"ecfa5332-9f62-4b49-b962-eb39fb148f16 is now active with session: cfe64458-c95d-41a9-b7f8-dd67d9fee52c","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.845069278Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.845077716Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"session_id":"cfe64458-c95d-41a9-b7f8-dd67d9fee52c"} {"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.845086655Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141} {"msg":"Activated sub_volume ecfa5332-9f62-4b49-b962-eb39fb148f16","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.845094443Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.845102402Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.845624394Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"task":"crutest"} 01/30 Write block 206 len 4096 data: 2 02/30 Read block 251 len 24576 03/30 Read block 593 len 20480 04/30 Read block 130 len 12288 05/30 Read block 112 len 24576 06/30 Write block 75 len 20480 data: 2 2 2 2 2 07/30 Read block 150 len 36864 08/30 Write block 8 len 20480 data: 2 2 2 2 2 09/30 Read block 509 len 16384 10/30 Flush 11/30 Flush 12/30 Write block 429 len 8192 data: 2 2 13/30 Read block 558 len 32768 14/30 Write block 134 len 32768 data: 2 2 2 2 2 2 2 2 15/30 Read block 223 len 40960 16/30 Write block 414 len 24576 data: 2 2 2 2 2 2 17/30 Write block 26 len 12288 data: 2 2 2 18/30 Write block 115 len 40960 data: 2 2 2 2 2 2 2 2 2 2 19/30 Write block 486 len 24576 data: 2 2 2 2 2 2 20/30 Write block 56 len 32768 data: 2 2 2 2 2 2 2 2 21/30 Read block 195 len 40960 22/30 Flush 23/30 Write block 256 len 32768 data: 2 2 2 2 2 2 2 2 24/30 Write block 557 len 40960 data: 2 2 2 2 2 2 2 2 2 2 25/30 Write block 376 len 36864 data: 2 2 2 2 2 2 2 2 2 26/30 Read block 24 len 40960 27/30 Read block 493 len 40960 28/30 Read block 527 len 8192 29/30 Read block 400 len 4096 30/30 Read block 214 len 36864 ---------------------------------------------------------------- Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Sent Done Done false 1001 Acked Read 6 Sent Done Done false 1002 Acked Read 5 Sent Done Done false 1003 Acked Read 3 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Write 5 Sent Done Done false 1006 Acked Read 9 Sent Done Done false 1007 Acked Write 5 Sent Done Done false 1008 Acked Read 4 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Read 8 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Write 3 Sent Done Done false 1017 Acked Write 10 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Read 10 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Read 9 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:21.854685307Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:21.854864173Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1141,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 1 2 2 1 2 2 F F F <--- 1 020-039 1 2 2 1 4 4 F F F <--- 2 040-059 1 2 2 1 4 4 F F F <--- 3 060-079 1 2 2 1 4 4 F F F <--- 4 080-099 1 1 1 1 1 1 F F F 5 100-119 1 2 2 1 4 4 F F F <--- 6 120-139 1 2 2 1 4 4 F F F <--- 7 140-159 1 2 2 1 4 4 F F F <--- 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 1 2 2 1 2 2 F F F <--- 11 220-239 1 1 1 1 1 1 F F F 12 240-259 1 1 1 1 1 1 F T T <--- 13 260-279 1 1 1 1 1 1 F T T <--- 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 1 1 1 1 1 1 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 1 1 1 1 1 1 F T T <--- 19 380-399 1 1 1 1 1 1 F T T <--- 20 400-419 1 2 2 1 4 4 F F F <--- 21 420-439 1 2 2 1 4 4 F F F <--- 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 1 2 2 1 4 4 F F F <--- 25 500-519 1 1 1 1 1 1 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 1 1 1 1 1 1 F T T <--- 28 560-579 1 1 1 1 1 1 F T T <--- 29 580-599 1 1 1 1 1 1 F F F Max gen: 2, Max flush: 4 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 1 Verifying data now No change after verify Loop: 1 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 2 2 2 4 4 4 F F F 2 040-059 2 2 2 4 4 4 F F F 3 060-079 2 2 2 4 4 4 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 2 2 2 4 4 4 F F F 6 120-139 2 2 2 4 4 4 F F F 7 140-159 2 2 2 4 4 4 F F F 8 160-179 1 1 1 1 1 1 F F F 9 180-199 1 1 1 1 1 1 F F F 10 200-219 2 2 2 2 2 2 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 3 3 3 5 5 5 F F F 13 260-279 3 3 3 5 5 5 F F F 14 280-299 1 1 1 1 1 1 F F F 15 300-319 1 1 1 1 1 1 F F F 16 320-339 1 1 1 1 1 1 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 3 3 3 5 5 5 F F F 19 380-399 3 3 3 5 5 5 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 2 2 2 4 4 4 F F F 22 440-459 1 1 1 1 1 1 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 2 2 2 4 4 4 F F F 25 500-519 1 1 1 1 1 1 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 3 3 3 5 5 5 F F F 28 560-579 3 3 3 5 5 5 F F F 29 580-599 1 1 1 1 1 1 F F F Max gen: 3, Max flush: 5 Begin loop 2 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.243493142Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.251755645Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252053659Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252082814Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252091252Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.25210275Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149} {"msg":"Crucible 6e484074-5565-4930-8285-c546cfa9f2c5 has session id: 2c995811-8e04-4fae-9377-4fce00596de6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.25242032Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"Upstairs opts: Upstairs UUID: 6e484074-5565-4930-8285-c546cfa9f2c5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252469211Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"Crucible stats registered with UUID: 6e484074-5565-4930-8285-c546cfa9f2c5","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252479809Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:26.252488938Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252631741Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"io task","client":"1","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252647838Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"io task","client":"0","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252657856Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"io task","client":"2","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252812577Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"task":"crutest"} {"msg":"The guest has requested activation with gen:4","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252824145Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149} {"msg":"6e484074-5565-4930-8285-c546cfa9f2c5 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252841821Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.25285059Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"client":"0","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252860228Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"client":"1","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252869696Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"client":"2","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252879244Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"io task","client":"2","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252889213Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"io task","client":"0","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.25290088Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"io task","client":"1","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"ds_connection connected from Ok(127.0.0.1:50835)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.25295399Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"io task","client":"1","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"ds_connection connected from Ok(127.0.0.1:52493)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252975926Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"io task","client":"0","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"ds_connection connected from Ok(127.0.0.1:63478)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.252987374Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"io task","client":"2","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.253646491Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"client":"0","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.253671806Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"client":"2","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.253775227Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"client":"1","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.253950994Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.253985067Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"[0]R flush_numbers[0..12]: [2, 4, 4, 4, 1, 4, 4, 4, 1, 1, 2, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254058004Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"downstairs","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"[0]R generation[0..12]: [2, 2, 2, 2, 1, 2, 2, 2, 1, 1, 2, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254070031Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"downstairs","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254081049Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"downstairs","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"[1]R flush_numbers[0..12]: [2, 4, 4, 4, 1, 4, 4, 4, 1, 1, 2, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254091297Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"downstairs","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"[1]R generation[0..12]: [2, 2, 2, 2, 1, 2, 2, 2, 1, 1, 2, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254101336Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"downstairs","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254114133Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"downstairs","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"[2]R flush_numbers[0..12]: [2, 4, 4, 4, 1, 4, 4, 4, 1, 1, 2, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254124391Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"downstairs","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"[2]R generation[0..12]: [2, 2, 2, 2, 1, 2, 2, 2, 1, 1, 2, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254134309Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"downstairs","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254144987Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"downstairs","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"Max found gen is 4","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254154246Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"downstairs","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"Generation requested: 4 >= found:4","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254163854Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"downstairs","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"Next flush: 6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254172952Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"downstairs","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.25418216Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"":"downstairs","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254192189Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254200777Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"client":"0","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254210705Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"client":"1","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254220563Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"client":"2","":"downstairs","session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254230561Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.25423961Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"6e484074-5565-4930-8285-c546cfa9f2c5 is now active with session: 2c995811-8e04-4fae-9377-4fce00596de6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254249498Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254258216Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"session_id":"2c995811-8e04-4fae-9377-4fce00596de6"} {"msg":"The guest has finished waiting for activation with:4","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254268334Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149} {"msg":"Activated sub_volume 6e484074-5565-4930-8285-c546cfa9f2c5","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254276143Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.254284181Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.25461335Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"task":"crutest"} 01/30 Write block 375 len 32768 data: 2 3 3 3 3 3 3 3 02/30 Flush 03/30 Write block 60 len 36864 data: 3 3 3 3 2 2 2 2 2 04/30 Flush 05/30 Write block 552 len 12288 data: 2 2 2 06/30 Write block 572 len 28672 data: 2 2 2 2 2 2 2 07/30 Flush 08/30 Write block 308 len 40960 data: 2 2 2 2 2 2 2 2 2 2 09/30 Read block 248 len 8192 10/30 Write block 587 len 40960 data: 2 2 2 2 2 2 2 2 2 2 11/30 Write block 283 len 36864 data: 2 2 2 2 2 2 2 2 2 12/30 Read block 408 len 40960 13/30 Write block 380 len 32768 data: 4 4 4 3 3 2 2 2 14/30 Write block 426 len 20480 data: 2 2 2 3 3 15/30 Write block 124 len 40960 data: 3 2 2 2 2 2 2 2 2 2 16/30 Write block 178 len 20480 data: 2 2 2 2 2 17/30 Write block 443 len 16384 data: 2 2 2 2 18/30 Read block 223 len 8192 19/30 Read block 156 len 16384 20/30 Read block 245 len 20480 21/30 Write block 26 len 16384 data: 3 3 3 2 22/30 Read block 329 len 16384 23/30 Flush 24/30 Write block 505 len 12288 data: 2 2 2 25/30 Read block 334 len 8192 26/30 Read block 301 len 16384 27/30 Write block 178 len 40960 data: 3 3 3 3 3 2 2 2 2 2 28/30 Read block 516 len 28672 29/30 Write block 306 len 8192 data: 2 2 30/30 Write block 255 len 20480 data: 2 3 3 3 3 ---------------------------------------------------------------- Crucible gen:4 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 9 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 3 Done Sent Done false 1005 Acked Write 7 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 10 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Write 10 Done Sent Done false 1010 Acked Write 9 Done Sent Done false 1011 Acked Read 10 Done Sent Done false 1012 Acked Write 8 Done Sent Done false 1013 Acked Write 5 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Write 4 Done Sent Done false 1021 Acked Read 4 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 3 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Read 4 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 29 2 33 Done 28 1 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:26.263462742Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:26.26363577Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1149,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 4 2 4 9 4 9 F F F <--- 2 040-059 2 2 2 4 4 4 F F F 3 060-079 4 2 4 7 4 7 F F F <--- 4 080-099 1 1 1 1 1 1 F F F 5 100-119 2 2 2 4 4 4 F F F 6 120-139 4 2 4 9 4 9 F F F <--- 7 140-159 2 2 2 4 4 4 F F F 8 160-179 4 1 4 9 1 9 T F T <--- 9 180-199 4 1 4 9 1 9 T F T <--- 10 200-219 2 2 2 2 2 2 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 3 3 3 5 5 5 T F T <--- 13 260-279 3 3 3 5 5 5 F F F 14 280-299 4 1 4 9 1 9 F F F <--- 15 300-319 4 1 4 9 1 9 T F T <--- 16 320-339 1 1 1 1 1 1 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 4 3 4 6 5 6 F T F <--- 19 380-399 4 3 4 9 5 9 F T F <--- 20 400-419 2 2 2 4 4 4 F F F 21 420-439 4 2 4 9 4 9 F F F <--- 22 440-459 4 1 4 9 1 9 F F F <--- 23 460-479 1 1 1 1 1 1 F F F 24 480-499 2 2 2 4 4 4 F F F 25 500-519 1 1 1 1 1 1 T F T <--- 26 520-539 1 1 1 1 1 1 F F F 27 540-559 4 3 4 8 5 8 F F F <--- 28 560-579 4 3 4 8 5 8 F F F <--- 29 580-599 4 1 4 9 1 9 F F F <--- Max gen: 4, Max flush: 9 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 2 Verifying data now No change after verify Loop: 2 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 4 4 4 9 9 9 F F F 2 040-059 2 2 2 4 4 4 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 2 2 2 4 4 4 F F F 6 120-139 4 4 4 9 9 9 F F F 7 140-159 2 2 2 4 4 4 F F F 8 160-179 5 5 5 10 10 10 F F F 9 180-199 5 5 5 10 10 10 F F F 10 200-219 2 2 2 2 2 2 F F F 11 220-239 1 1 1 1 1 1 F F F 12 240-259 5 5 5 10 10 10 F F F 13 260-279 3 3 3 5 5 5 F F F 14 280-299 4 4 4 9 9 9 F F F 15 300-319 5 5 5 10 10 10 F F F 16 320-339 1 1 1 1 1 1 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 4 4 4 6 6 6 F F F 19 380-399 4 4 4 9 9 9 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 4 4 4 9 9 9 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 2 2 2 4 4 4 F F F 25 500-519 5 5 5 10 10 10 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 4 4 4 8 8 8 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 4 4 4 9 9 9 F F F Max gen: 5, Max flush: 10 Begin loop 3 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.51273811Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:30.519803087Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.519857276Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.526745156Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527008387Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527033822Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527042211Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527053679Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157} {"msg":"Crucible 17d8ecc9-3089-4c14-892b-e864f3ca34d6 has session id: f6b59689-26fe-4e84-b701-cea78189db1a","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527369959Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"Upstairs opts: Upstairs UUID: 17d8ecc9-3089-4c14-892b-e864f3ca34d6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527398164Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"Crucible stats registered with UUID: 17d8ecc9-3089-4c14-892b-e864f3ca34d6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527411412Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:30.52742053Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527556814Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"io task","client":"1","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527586959Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"io task","client":"0","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527597897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"io task","client":"2","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527711605Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"task":"crutest"} {"msg":"The guest has requested activation with gen:6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527729932Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157} {"msg":"17d8ecc9-3089-4c14-892b-e864f3ca34d6 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527745779Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527754487Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"client":"0","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527764036Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"client":"1","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527773694Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"client":"2","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527783132Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"io task","client":"2","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.52779298Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"io task","client":"0","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527803108Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"io task","client":"1","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"ds_connection connected from Ok(127.0.0.1:42654)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527870456Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"io task","client":"2","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"ds_connection connected from Ok(127.0.0.1:41195)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527895851Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"io task","client":"1","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"ds_connection connected from Ok(127.0.0.1:51068)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.527909078Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"io task","client":"0","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.528585342Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"client":"1","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.528606818Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"client":"2","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.528617646Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"client":"0","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.528881956Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.528903882Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"[0]R flush_numbers[0..12]: [2, 9, 4, 7, 1, 4, 9, 4, 10, 10, 2, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.52891315Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"downstairs","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"[0]R generation[0..12]: [2, 4, 2, 4, 1, 2, 4, 2, 5, 5, 2, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.528923498Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"downstairs","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.528934346Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"downstairs","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"[1]R flush_numbers[0..12]: [2, 9, 4, 7, 1, 4, 9, 4, 10, 10, 2, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.528944405Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"downstairs","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"[1]R generation[0..12]: [2, 4, 2, 4, 1, 2, 4, 2, 5, 5, 2, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.528954423Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"downstairs","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.528965261Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"downstairs","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"[2]R flush_numbers[0..12]: [2, 9, 4, 7, 1, 4, 9, 4, 10, 10, 2, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.528975239Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"downstairs","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"[2]R generation[0..12]: [2, 4, 2, 4, 1, 2, 4, 2, 5, 5, 2, 1]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.528985287Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"downstairs","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529005293Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"downstairs","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"Max found gen is 6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529014731Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"downstairs","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"Generation requested: 6 >= found:6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.52902442Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"downstairs","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"Next flush: 11","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529033838Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"downstairs","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529043136Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"":"downstairs","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529052684Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529061413Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"client":"0","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529070951Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"client":"1","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529080519Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"client":"2","":"downstairs","session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529090227Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529099116Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"17d8ecc9-3089-4c14-892b-e864f3ca34d6 is now active with session: f6b59689-26fe-4e84-b701-cea78189db1a","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529108834Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529117512Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"session_id":"f6b59689-26fe-4e84-b701-cea78189db1a"} {"msg":"The guest has finished waiting for activation with:6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.52912641Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157} {"msg":"Activated sub_volume 17d8ecc9-3089-4c14-892b-e864f3ca34d6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529134189Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529142158Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.529467637Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"task":"crutest"} 01/30 Write block 112 len 40960 data: 2 2 2 3 3 3 3 3 3 3 02/30 Read block 403 len 28672 03/30 Read block 244 len 32768 04/30 Read block 177 len 28672 05/30 Write block 383 len 8192 data: 4 4 06/30 Read block 126 len 16384 07/30 Flush 08/30 Write block 364 len 8192 data: 2 2 09/30 Write block 113 len 16384 data: 3 3 4 4 10/30 Read block 526 len 4096 11/30 Read block 70 len 16384 12/30 Write block 41 len 24576 data: 2 2 2 2 2 2 13/30 Read block 374 len 12288 14/30 Read block 81 len 16384 15/30 Read block 325 len 12288 16/30 Write block 46 len 28672 data: 3 2 2 2 2 2 2 17/30 Read block 108 len 4096 18/30 Write block 374 len 40960 data: 2 3 4 4 4 4 5 5 5 5 19/30 Write block 227 len 8192 data: 2 2 20/30 Read block 31 len 16384 21/30 Read block 480 len 40960 22/30 Read block 151 len 16384 23/30 Read block 156 len 24576 24/30 Write block 319 len 16384 data: 2 2 2 2 25/30 Write block 172 len 12288 data: 2 2 2 26/30 Flush 27/30 Write block 130 len 16384 data: 3 3 3 3 28/30 Read block 418 len 28672 29/30 Read block 124 len 20480 30/30 Read block 320 len 36864 ---------------------------------------------------------------- Crucible gen:6 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Read 7 Done Done Done false 1004 Acked Write 2 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Read 1 Sent Done Done false 1010 Acked Read 4 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Read 4 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Read 1 Sent Done Done false 1017 Acked Write 10 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Read 4 Sent Done Done false 1020 Acked Read 10 Sent Done Done false 1021 Acked Read 4 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Flush 0 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Read 9 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 0 1 27 Done 4 30 29 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1025)) Some(JobId(1025)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:30.538148591Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:30.538319159Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1157,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 4 4 4 9 9 9 F F F 2 040-059 2 6 6 4 12 12 F F F <--- 3 060-079 4 4 4 7 7 7 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 2 6 6 4 12 12 T F F <--- 6 120-139 4 6 6 9 11 11 T T T <--- 7 140-159 2 2 2 4 4 4 F F F 8 160-179 5 6 6 10 12 12 F F F <--- 9 180-199 5 5 5 10 10 10 F F F 10 200-219 2 2 2 2 2 2 F F F 11 220-239 1 6 6 1 12 12 F F F <--- 12 240-259 5 5 5 10 10 10 F F F 13 260-279 3 3 3 5 5 5 F F F 14 280-299 4 4 4 9 9 9 F F F 15 300-319 5 6 6 10 12 12 F F F <--- 16 320-339 1 6 6 1 12 12 F F F <--- 17 340-359 1 1 1 1 1 1 F F F 18 360-379 4 6 6 6 12 12 F F F <--- 19 380-399 4 6 6 9 12 12 F F F <--- 20 400-419 2 2 2 4 4 4 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 4 4 4 9 9 9 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 2 2 2 4 4 4 F F F 25 500-519 5 5 5 10 10 10 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 4 4 4 8 8 8 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 4 4 4 9 9 9 F F F Max gen: 6, Max flush: 12 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 3 Verifying data now No change after verify Loop: 3 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 4 4 4 9 9 9 F F F 2 040-059 6 6 6 12 12 12 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 1 1 1 1 1 1 F F F 5 100-119 6 6 6 12 12 12 F F F 6 120-139 7 7 7 13 13 13 F F F 7 140-159 2 2 2 4 4 4 F F F 8 160-179 6 6 6 12 12 12 F F F 9 180-199 5 5 5 10 10 10 F F F 10 200-219 2 2 2 2 2 2 F F F 11 220-239 6 6 6 12 12 12 F F F 12 240-259 5 5 5 10 10 10 F F F 13 260-279 3 3 3 5 5 5 F F F 14 280-299 4 4 4 9 9 9 F F F 15 300-319 6 6 6 12 12 12 F F F 16 320-339 6 6 6 12 12 12 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 6 6 6 12 12 12 F F F 19 380-399 6 6 6 12 12 12 F F F 20 400-419 2 2 2 4 4 4 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 4 4 4 9 9 9 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 2 2 2 4 4 4 F F F 25 500-519 5 5 5 10 10 10 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 4 4 4 8 8 8 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 4 4 4 9 9 9 F F F Max gen: 7, Max flush: 13 Begin loop 4 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.774655079Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:34.781771817Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.781824087Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.788524362Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.788789332Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.788816797Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.788825256Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.788836644Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165} {"msg":"Crucible 25d7cce4-5b35-41f3-9e95-74fcb436f784 has session id: eb3f4743-fd90-4379-bc39-4d18799ba589","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789193327Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"Upstairs opts: Upstairs UUID: 25d7cce4-5b35-41f3-9e95-74fcb436f784, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789216072Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"Crucible stats registered with UUID: 25d7cce4-5b35-41f3-9e95-74fcb436f784","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789226031Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:34.789234799Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789381801Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"io task","client":"1","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789397738Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"io task","client":"0","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789407657Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"io task","client":"2","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789555939Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"task":"crutest"} {"msg":"The guest has requested activation with gen:8","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789567457Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165} {"msg":"25d7cce4-5b35-41f3-9e95-74fcb436f784 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.78960063Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789610379Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"client":"0","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789620087Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"client":"1","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789629485Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"client":"2","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789638873Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"io task","client":"2","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789648661Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"io task","client":"0","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.7896585Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"io task","client":"1","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"ds_connection connected from Ok(127.0.0.1:34676)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789783866Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"io task","client":"1","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"ds_connection connected from Ok(127.0.0.1:37183)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789809881Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"io task","client":"0","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"ds_connection connected from Ok(127.0.0.1:60228)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.789821049Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"io task","client":"2","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790332473Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"client":"2","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790354879Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"client":"1","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790531776Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"client":"0","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790627698Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790663481Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"[0]R flush_numbers[0..12]: [2, 9, 12, 7, 1, 12, 13, 4, 12, 10, 2, 12]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790787698Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"downstairs","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"[0]R generation[0..12]: [2, 4, 6, 4, 1, 6, 7, 2, 6, 5, 2, 6]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790798516Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"downstairs","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790809574Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"downstairs","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"[1]R flush_numbers[0..12]: [2, 9, 12, 7, 1, 12, 13, 4, 12, 10, 2, 12]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790819772Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"downstairs","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"[1]R generation[0..12]: [2, 4, 6, 4, 1, 6, 7, 2, 6, 5, 2, 6]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.79082987Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"downstairs","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790840918Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"downstairs","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"[2]R flush_numbers[0..12]: [2, 9, 12, 7, 1, 12, 13, 4, 12, 10, 2, 12]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790851176Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"downstairs","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"[2]R generation[0..12]: [2, 4, 6, 4, 1, 6, 7, 2, 6, 5, 2, 6]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790861304Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"downstairs","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790872052Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"downstairs","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"Max found gen is 8","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790881301Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"downstairs","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"Generation requested: 8 >= found:8","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790890909Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"downstairs","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"Next flush: 14","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790900147Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"downstairs","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790909525Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"":"downstairs","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790920303Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790928882Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"client":"0","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.79093842Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"client":"1","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790947908Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"client":"2","":"downstairs","session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790957606Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790966625Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"25d7cce4-5b35-41f3-9e95-74fcb436f784 is now active with session: eb3f4743-fd90-4379-bc39-4d18799ba589","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790976403Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.790984971Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"session_id":"eb3f4743-fd90-4379-bc39-4d18799ba589"} {"msg":"The guest has finished waiting for activation with:8","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.79099385Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165} {"msg":"Activated sub_volume 25d7cce4-5b35-41f3-9e95-74fcb436f784","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.791001708Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.791009817Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.791329337Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"task":"crutest"} 01/30 Write block 328 len 12288 data: 2 2 2 02/30 Write block 333 len 20480 data: 2 2 2 2 2 03/30 Read block 317 len 32768 04/30 Flush 05/30 Write block 394 len 40960 data: 2 2 2 2 2 2 2 2 2 2 06/30 Read block 253 len 12288 07/30 Write block 183 len 16384 data: 3 3 3 3 08/30 Write block 454 len 4096 data: 2 09/30 Read block 89 len 8192 10/30 Write block 310 len 16384 data: 3 3 3 3 11/30 Flush 12/30 Write block 482 len 24576 data: 2 2 2 2 3 3 13/30 Read block 430 len 20480 14/30 Read block 68 len 12288 15/30 Write block 84 len 36864 data: 2 2 2 2 2 2 2 2 2 16/30 Write block 144 len 8192 data: 2 2 17/30 Read block 272 len 8192 18/30 Flush 19/30 Write block 451 len 4096 data: 2 20/30 Read block 532 len 12288 21/30 Read block 3 len 24576 22/30 Flush 23/30 Write block 488 len 36864 data: 3 3 3 3 2 2 2 2 2 24/30 Read block 492 len 8192 25/30 Read block 56 len 20480 26/30 Write block 555 len 12288 data: 2 2 3 27/30 Read block 212 len 36864 28/30 Read block 367 len 20480 29/30 Write block 545 len 36864 data: 2 2 2 2 2 2 2 3 3 30/30 Read block 309 len 40960 ---------------------------------------------------------------- Crucible gen:8 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 10 Sent Done Done false 1005 Acked Read 3 Sent Done Done false 1006 Acked Write 4 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Read 2 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 6 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 9 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Read 9 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Write 9 Sent Done Sent false 1029 Acked Read 10 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 0 2 28 Done 0 26 24 50 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1003)) Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:34.800073439Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:34.800255595Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1165,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 4 4 4 9 9 9 F F F 2 040-059 6 6 6 12 12 12 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 1 8 8 1 16 16 F F F <--- 5 100-119 6 6 6 12 12 12 F F F 6 120-139 7 7 7 13 13 13 F F F 7 140-159 2 8 8 4 16 16 F F F <--- 8 160-179 6 6 6 12 12 12 F F F 9 180-199 5 8 8 10 15 15 F F F <--- 10 200-219 2 2 2 2 2 2 F F F 11 220-239 6 6 6 12 12 12 F F F 12 240-259 5 5 5 10 10 10 F F F 13 260-279 3 3 3 5 5 5 F F F 14 280-299 4 4 4 9 9 9 F F F 15 300-319 6 8 8 12 15 15 F F F <--- 16 320-339 8 8 8 14 14 14 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 6 6 6 12 12 12 F F F 19 380-399 6 8 8 12 15 15 F F F <--- 20 400-419 2 8 8 4 15 15 F F F <--- 21 420-439 4 4 4 9 9 9 F F F 22 440-459 4 8 8 9 17 17 F F F <--- 23 460-479 1 1 1 1 1 1 F F F 24 480-499 2 8 8 4 16 16 F T T <--- 25 500-519 5 5 5 10 10 10 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 4 4 4 8 8 8 F T T <--- 28 560-579 4 4 4 8 8 8 F F F 29 580-599 4 4 4 9 9 9 F F F Max gen: 8, Max flush: 17 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 4 Verifying data now No change after verify Loop: 4 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 4 4 4 9 9 9 F F F 2 040-059 6 6 6 12 12 12 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 8 8 8 16 16 16 F F F 5 100-119 6 6 6 12 12 12 F F F 6 120-139 7 7 7 13 13 13 F F F 7 140-159 8 8 8 16 16 16 F F F 8 160-179 6 6 6 12 12 12 F F F 9 180-199 8 8 8 15 15 15 F F F 10 200-219 2 2 2 2 2 2 F F F 11 220-239 6 6 6 12 12 12 F F F 12 240-259 5 5 5 10 10 10 F F F 13 260-279 3 3 3 5 5 5 F F F 14 280-299 4 4 4 9 9 9 F F F 15 300-319 8 8 8 15 15 15 F F F 16 320-339 8 8 8 14 14 14 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 6 6 6 12 12 12 F F F 19 380-399 8 8 8 15 15 15 F F F 20 400-419 8 8 8 15 15 15 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 8 8 8 17 17 17 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 9 9 9 18 18 18 F F F 25 500-519 5 5 5 10 10 10 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 9 9 9 18 18 18 F F F 28 560-579 4 4 4 8 8 8 F F F 29 580-599 4 4 4 9 9 9 F F F Max gen: 9, Max flush: 18 Begin loop 5 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.04137083Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:39.048535319Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.048584529Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.055384086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.055659474Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.05568514Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.055696497Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.055707885Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173} {"msg":"Crucible 348ffe89-bb8b-41c8-9d14-61e0fedb901a has session id: 45b0bab6-1edd-47b5-9447-3838b8491694","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056038563Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"Upstairs opts: Upstairs UUID: 348ffe89-bb8b-41c8-9d14-61e0fedb901a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056062439Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"Crucible stats registered with UUID: 348ffe89-bb8b-41c8-9d14-61e0fedb901a","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056072097Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:39.056080965Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056225088Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"io task","client":"0","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056241965Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"io task","client":"1","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056251963Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"io task","client":"2","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056400396Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"task":"crutest"} {"msg":"The guest has requested activation with gen:10","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056412783Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173} {"msg":"348ffe89-bb8b-41c8-9d14-61e0fedb901a active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056436019Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056445197Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"client":"0","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056454695Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"client":"1","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056464154Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"client":"2","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056475132Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"io task","client":"2","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.05648485Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"io task","client":"0","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056494618Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"io task","client":"1","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"ds_connection connected from Ok(127.0.0.1:64923)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.05653599Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"io task","client":"2","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"ds_connection connected from Ok(127.0.0.1:62762)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056546978Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"io task","client":"1","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"ds_connection connected from Ok(127.0.0.1:41746)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.056557286Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"io task","client":"0","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.05723036Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"client":"1","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057243138Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"client":"2","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057253846Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"client":"0","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057528744Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057538222Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"[0]R flush_numbers[0..12]: [2, 9, 12, 7, 16, 12, 13, 16, 12, 15, 2, 12]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057578675Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"downstairs","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"[0]R generation[0..12]: [2, 4, 6, 4, 8, 6, 7, 8, 6, 8, 2, 6]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057589413Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"downstairs","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057600321Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"downstairs","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"[1]R flush_numbers[0..12]: [2, 9, 12, 7, 16, 12, 13, 16, 12, 15, 2, 12]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057612389Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"downstairs","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"[1]R generation[0..12]: [2, 4, 6, 4, 8, 6, 7, 8, 6, 8, 2, 6]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057622427Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"downstairs","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057633205Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"downstairs","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"[2]R flush_numbers[0..12]: [2, 9, 12, 7, 16, 12, 13, 16, 12, 15, 2, 12]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057643453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"downstairs","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"[2]R generation[0..12]: [2, 4, 6, 4, 8, 6, 7, 8, 6, 8, 2, 6]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057653451Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"downstairs","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057664199Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"downstairs","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"Max found gen is 10","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057673517Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"downstairs","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"Generation requested: 10 >= found:10","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057683205Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"downstairs","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"Next flush: 19","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057692294Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"downstairs","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057701762Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"":"downstairs","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.05771138Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057720118Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"client":"0","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057729787Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"client":"1","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057739365Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"client":"2","":"downstairs","session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057749143Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057758181Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"348ffe89-bb8b-41c8-9d14-61e0fedb901a is now active with session: 45b0bab6-1edd-47b5-9447-3838b8491694","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057769489Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057778138Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"session_id":"45b0bab6-1edd-47b5-9447-3838b8491694"} {"msg":"The guest has finished waiting for activation with:10","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057787026Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173} {"msg":"Activated sub_volume 348ffe89-bb8b-41c8-9d14-61e0fedb901a","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057795244Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.057803393Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.058102817Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"task":"crutest"} 01/30 Write block 84 len 12288 data: 3 3 3 02/30 Write block 441 len 28672 data: 2 2 3 3 3 3 2 03/30 Read block 96 len 12288 04/30 Read block 160 len 32768 05/30 Read block 139 len 4096 06/30 Write block 317 len 32768 data: 3 2 3 3 3 3 2 2 07/30 Read block 587 len 12288 08/30 Read block 313 len 32768 09/30 Write block 304 len 16384 data: 2 2 3 3 10/30 Write block 219 len 16384 data: 2 2 2 2 11/30 Read block 565 len 24576 12/30 Write block 320 len 8192 data: 4 4 13/30 Read block 283 len 40960 14/30 Read block 389 len 4096 15/30 Write block 371 len 20480 data: 2 2 2 3 4 16/30 Read block 39 len 4096 17/30 Flush 18/30 Write block 120 len 8192 data: 4 4 19/30 Read block 450 len 40960 20/30 Read block 213 len 8192 21/30 Write block 515 len 16384 data: 2 2 2 2 22/30 Write block 306 len 36864 data: 4 4 3 3 4 4 4 4 3 23/30 Read block 55 len 4096 24/30 Write block 123 len 32768 data: 3 4 3 3 3 3 3 4 25/30 Read block 466 len 4096 26/30 Read block 73 len 40960 27/30 Read block 299 len 16384 28/30 Read block 279 len 40960 29/30 Write block 578 len 4096 data: 3 30/30 Read block 45 len 20480 ---------------------------------------------------------------- Crucible gen:10 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Write 8 Sent Done Done false 1006 Acked Read 3 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Write 4 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Read 10 Sent Done Done false 1013 Acked Read 1 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Read 1 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 2 Sent Done Done false 1018 Acked Read 10 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Write 4 Sent Done Done false 1021 Acked Write 9 Sent Done Done false 1022 Acked Read 1 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Read 10 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Write 1 Sent Done Done false 1029 Acked Read 5 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 0 1 27 Done 4 30 29 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:39.066717703Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:39.066900999Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1173,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 4 4 4 9 9 9 F F F 2 040-059 6 6 6 12 12 12 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 8 10 10 16 19 19 T F F <--- 5 100-119 6 6 6 12 12 12 F F F 6 120-139 7 7 7 13 13 13 F T T <--- 7 140-159 8 8 8 16 16 16 F F F 8 160-179 6 6 6 12 12 12 F F F 9 180-199 8 8 8 15 15 15 F F F 10 200-219 2 10 10 2 19 19 F F F <--- 11 220-239 6 10 10 12 19 19 F F F <--- 12 240-259 5 5 5 10 10 10 F F F 13 260-279 3 3 3 5 5 5 F F F 14 280-299 4 4 4 9 9 9 F F F 15 300-319 8 10 10 15 19 19 F T T <--- 16 320-339 8 10 10 14 19 19 F F F <--- 17 340-359 1 1 1 1 1 1 F F F 18 360-379 6 10 10 12 19 19 F F F <--- 19 380-399 8 8 8 15 15 15 F F F 20 400-419 8 8 8 15 15 15 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 8 10 10 17 19 19 T F F <--- 23 460-479 1 1 1 1 1 1 F F F 24 480-499 9 9 9 18 18 18 F F F 25 500-519 5 5 5 10 10 10 F T T <--- 26 520-539 1 1 1 1 1 1 F F F 27 540-559 9 9 9 18 18 18 F F F 28 560-579 4 4 4 8 8 8 F T T <--- 29 580-599 4 4 4 9 9 9 F F F Max gen: 10, Max flush: 19 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 5 Verifying data now No change after verify Loop: 5 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 F F F 1 020-039 4 4 4 9 9 9 F F F 2 040-059 6 6 6 12 12 12 F F F 3 060-079 4 4 4 7 7 7 F F F 4 080-099 10 10 10 19 19 19 F F F 5 100-119 6 6 6 12 12 12 F F F 6 120-139 11 11 11 20 20 20 F F F 7 140-159 8 8 8 16 16 16 F F F 8 160-179 6 6 6 12 12 12 F F F 9 180-199 8 8 8 15 15 15 F F F 10 200-219 10 10 10 19 19 19 F F F 11 220-239 10 10 10 19 19 19 F F F 12 240-259 5 5 5 10 10 10 F F F 13 260-279 3 3 3 5 5 5 F F F 14 280-299 4 4 4 9 9 9 F F F 15 300-319 11 11 11 20 20 20 F F F 16 320-339 10 10 10 19 19 19 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 10 10 10 19 19 19 F F F 19 380-399 8 8 8 15 15 15 F F F 20 400-419 8 8 8 15 15 15 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 10 10 10 19 19 19 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 9 9 9 18 18 18 F F F 25 500-519 11 11 11 20 20 20 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 9 9 9 18 18 18 F F F 28 560-579 11 11 11 20 20 20 F F F 29 580-599 4 4 4 9 9 9 F F F Max gen: 11, Max flush: 20 Begin loop 6 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.30729035Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.315484336Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.315784809Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.315812424Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.315821013Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.315832341Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181} {"msg":"Crucible 3c9a8f05-3d45-4072-becc-022d63b1d371 has session id: 90465c76-d5b7-4e48-963e-e260b917960d","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316191233Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"Upstairs opts: Upstairs UUID: 3c9a8f05-3d45-4072-becc-022d63b1d371, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316214639Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"Crucible stats registered with UUID: 3c9a8f05-3d45-4072-becc-022d63b1d371","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316226567Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:43.316235425Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316399134Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"io task","client":"2","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316429699Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"io task","client":"0","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316440607Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"io task","client":"1","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316550776Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"task":"crutest"} {"msg":"The guest has requested activation with gen:12","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316562554Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181} {"msg":"3c9a8f05-3d45-4072-becc-022d63b1d371 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316578341Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316586999Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"client":"0","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316596477Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"client":"1","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316605906Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"client":"2","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316615284Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"io task","client":"2","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316625112Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"io task","client":"0","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.31663494Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"io task","client":"1","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"ds_connection connected from Ok(127.0.0.1:54979)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.31679723Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"io task","client":"0","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"ds_connection connected from Ok(127.0.0.1:55917)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316854779Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"io task","client":"1","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"ds_connection connected from Ok(127.0.0.1:47840)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.316867517Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"io task","client":"2","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317493929Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"client":"1","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317517175Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"client":"2","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317532742Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"client":"0","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317828907Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317838885Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"[0]R flush_numbers[0..12]: [2, 9, 12, 7, 19, 12, 20, 16, 12, 15, 19, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317877008Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"downstairs","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"[0]R generation[0..12]: [2, 4, 6, 4, 10, 6, 11, 8, 6, 8, 10, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317887926Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"downstairs","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317898963Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"downstairs","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"[1]R flush_numbers[0..12]: [2, 9, 12, 7, 19, 12, 20, 16, 12, 15, 19, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317909292Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"downstairs","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"[1]R generation[0..12]: [2, 4, 6, 4, 10, 6, 11, 8, 6, 8, 10, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.31791951Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"downstairs","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317930628Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"downstairs","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"[2]R flush_numbers[0..12]: [2, 9, 12, 7, 19, 12, 20, 16, 12, 15, 19, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317940836Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"downstairs","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"[2]R generation[0..12]: [2, 4, 6, 4, 10, 6, 11, 8, 6, 8, 10, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317950954Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"downstairs","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317961722Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"downstairs","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"Max found gen is 12","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317973619Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"downstairs","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"Generation requested: 12 >= found:12","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317983238Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"downstairs","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"Next flush: 21","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.317992566Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"downstairs","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.318001974Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"":"downstairs","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.318011822Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.318020591Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"client":"0","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.318030129Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"client":"1","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.318039737Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"client":"2","":"downstairs","session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.318049495Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.318058554Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"3c9a8f05-3d45-4072-becc-022d63b1d371 is now active with session: 90465c76-d5b7-4e48-963e-e260b917960d","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.318068342Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.31807697Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"session_id":"90465c76-d5b7-4e48-963e-e260b917960d"} {"msg":"The guest has finished waiting for activation with:12","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.318085938Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181} {"msg":"Activated sub_volume 3c9a8f05-3d45-4072-becc-022d63b1d371","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.318093857Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.318101945Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.318426005Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"task":"crutest"} 01/30 Flush 02/30 Write block 303 len 36864 data: 2 3 3 5 5 4 4 5 5 03/30 Flush 04/30 Write block 122 len 20480 data: 3 4 5 4 4 05/30 Write block 503 len 12288 data: 2 2 3 06/30 Flush 07/30 Write block 267 len 12288 data: 2 2 2 08/30 Flush 09/30 Write block 28 len 4096 data: 4 10/30 Read block 467 len 32768 11/30 Read block 183 len 32768 12/30 Read block 108 len 40960 13/30 Write block 189 len 24576 data: 2 2 2 2 2 2 14/30 Read block 86 len 4096 15/30 Write block 29 len 28672 data: 3 2 2 2 2 2 2 16/30 Read block 36 len 12288 17/30 Write block 266 len 20480 data: 2 3 3 3 2 18/30 Read block 590 len 36864 19/30 Write block 147 len 4096 data: 2 20/30 Write block 332 len 4096 data: 2 21/30 Read block 357 len 36864 22/30 Read block 143 len 36864 23/30 Read block 205 len 40960 24/30 Write block 314 len 40960 data: 4 3 3 4 3 4 5 5 4 3 25/30 Write block 189 len 32768 data: 3 3 3 3 3 3 2 2 26/30 Flush 27/30 Write block 10 len 28672 data: 3 3 3 2 2 2 2 28/30 Read block 300 len 20480 29/30 Read block 498 len 32768 30/30 Write block 62 len 36864 data: 4 4 3 3 3 3 3 2 2 ---------------------------------------------------------------- Crucible gen:12 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 9 Done Done Done false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 5 Done Done Sent false 1004 Acked Write 3 Done Done Sent false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 3 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Read 10 Done Done Sent false 1012 Acked Write 6 Done Done Sent false 1013 Acked Read 1 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Read 9 Done Done Sent false 1018 Acked Write 1 Done Done Sent false 1019 Acked Write 1 Done Done Sent false 1020 Acked Read 9 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Read 10 Done Done Sent false 1023 Acked Write 10 Done Done Sent false 1024 Acked Write 8 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Read 5 Done Done Sent false 1028 Acked Read 8 Done Done Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 28 30 Done 28 28 1 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:43.327238404Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:43.327425349Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1181,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 2 2 2 2 2 2 T T F <--- 1 020-039 12 12 4 25 25 9 F F F <--- 2 040-059 6 6 6 12 12 12 F F F 3 060-079 4 4 4 7 7 7 T T F <--- 4 080-099 10 10 10 19 19 19 F F F 5 100-119 6 6 6 12 12 12 F F F 6 120-139 12 12 11 23 23 20 F F F <--- 7 140-159 12 12 8 25 25 16 F F F <--- 8 160-179 6 6 6 12 12 12 F F F 9 180-199 12 12 8 25 25 15 F F F <--- 10 200-219 10 10 10 19 19 19 F F F 11 220-239 10 10 10 19 19 19 F F F 12 240-259 5 5 5 10 10 10 F F F 13 260-279 12 12 3 25 25 5 F F F <--- 14 280-299 4 4 4 9 9 9 F F F 15 300-319 12 12 11 25 25 20 F F T <--- 16 320-339 12 12 10 25 25 19 F F F <--- 17 340-359 1 1 1 1 1 1 F F F 18 360-379 10 10 10 19 19 19 F F F 19 380-399 8 8 8 15 15 15 F F F 20 400-419 8 8 8 15 15 15 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 10 10 10 19 19 19 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 9 9 9 18 18 18 F F F 25 500-519 12 12 11 23 23 20 F F F <--- 26 520-539 1 1 1 1 1 1 F F F 27 540-559 9 9 9 18 18 18 F F F 28 560-579 11 11 11 20 20 20 F F F 29 580-599 4 4 4 9 9 9 F F F Max gen: 12, Max flush: 25 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 6 Verifying data now No change after verify Loop: 6 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 26 26 26 F F F 1 020-039 12 12 12 25 25 25 F F F 2 040-059 6 6 6 12 12 12 F F F 3 060-079 13 13 13 26 26 26 F F F 4 080-099 10 10 10 19 19 19 F F F 5 100-119 6 6 6 12 12 12 F F F 6 120-139 12 12 12 23 23 23 F F F 7 140-159 12 12 12 25 25 25 F F F 8 160-179 6 6 6 12 12 12 F F F 9 180-199 12 12 12 25 25 25 F F F 10 200-219 10 10 10 19 19 19 F F F 11 220-239 10 10 10 19 19 19 F F F 12 240-259 5 5 5 10 10 10 F F F 13 260-279 12 12 12 25 25 25 F F F 14 280-299 4 4 4 9 9 9 F F F 15 300-319 12 12 12 25 25 25 F F F 16 320-339 12 12 12 25 25 25 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 10 10 10 19 19 19 F F F 19 380-399 8 8 8 15 15 15 F F F 20 400-419 8 8 8 15 15 15 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 10 10 10 19 19 19 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 9 9 9 18 18 18 F F F 25 500-519 12 12 12 23 23 23 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 9 9 9 18 18 18 F F F 28 560-579 11 11 11 20 20 20 F F F 29 580-599 4 4 4 9 9 9 F F F Max gen: 13, Max flush: 26 Begin loop 7 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.564505551Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.572587897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.572887791Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.572915686Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.572924204Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.572935642Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189} {"msg":"Crucible da2b03d5-dba6-420f-a9f7-41d66bce4755 has session id: 3cdfd5ff-3303-4191-9811-e90c3a20ff60","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573342806Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"Upstairs opts: Upstairs UUID: da2b03d5-dba6-420f-a9f7-41d66bce4755, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.57337054Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"Crucible stats registered with UUID: da2b03d5-dba6-420f-a9f7-41d66bce4755","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573381538Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:47.573390877Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.57353304Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"io task","client":"0","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573562874Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"io task","client":"1","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573574102Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"io task","client":"2","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573716666Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"task":"crutest"} {"msg":"The guest has requested activation with gen:14","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573732283Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189} {"msg":"da2b03d5-dba6-420f-a9f7-41d66bce4755 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573757898Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573767216Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"client":"0","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573776724Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"client":"1","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573786232Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"client":"2","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573795631Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"io task","client":"2","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573805399Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"io task","client":"0","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573829744Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"io task","client":"1","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"ds_connection connected from Ok(127.0.0.1:34973)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573880185Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"io task","client":"1","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"ds_connection connected from Ok(127.0.0.1:60051)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573891133Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"io task","client":"0","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"ds_connection connected from Ok(127.0.0.1:64485)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.573901231Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"io task","client":"2","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.574502038Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"client":"0","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.574636093Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"client":"1","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.574656889Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"client":"2","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.574791514Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.574913391Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"[0]R flush_numbers[0..12]: [26, 25, 12, 26, 19, 12, 23, 25, 12, 25, 19, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.57497332Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"downstairs","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"[0]R generation[0..12]: [13, 12, 6, 13, 10, 6, 12, 12, 6, 12, 10, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.574988167Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"downstairs","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.574999375Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"downstairs","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"[1]R flush_numbers[0..12]: [26, 25, 12, 26, 19, 12, 23, 25, 12, 25, 19, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575009803Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"downstairs","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"[1]R generation[0..12]: [13, 12, 6, 13, 10, 6, 12, 12, 6, 12, 10, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575020161Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"downstairs","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575031029Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"downstairs","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"[2]R flush_numbers[0..12]: [26, 25, 12, 26, 19, 12, 23, 25, 12, 25, 19, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575041277Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"downstairs","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"[2]R generation[0..12]: [13, 12, 6, 13, 10, 6, 12, 12, 6, 12, 10, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575051386Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"downstairs","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575062304Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"downstairs","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"Max found gen is 14","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575071572Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"downstairs","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"Generation requested: 14 >= found:14","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.57508122Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"downstairs","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"Next flush: 27","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575090458Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"downstairs","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575099647Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"":"downstairs","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575109585Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575118413Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"client":"0","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575129331Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"client":"1","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575139189Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"client":"2","":"downstairs","session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575149237Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575158356Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"da2b03d5-dba6-420f-a9f7-41d66bce4755 is now active with session: 3cdfd5ff-3303-4191-9811-e90c3a20ff60","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575168254Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575176962Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"session_id":"3cdfd5ff-3303-4191-9811-e90c3a20ff60"} {"msg":"The guest has finished waiting for activation with:14","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.57518601Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189} {"msg":"Activated sub_volume da2b03d5-dba6-420f-a9f7-41d66bce4755","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575193949Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575201887Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.575530226Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"task":"crutest"} 01/30 Flush 02/30 Write block 565 len 12288 data: 3 3 2 03/30 Write block 598 len 8192 data: 2 2 04/30 Read block 408 len 8192 05/30 Read block 323 len 16384 06/30 Read block 523 len 4096 07/30 Read block 195 len 40960 08/30 Write block 191 len 12288 data: 4 4 4 09/30 Write block 94 len 16384 data: 2 2 2 2 10/30 Read block 546 len 28672 11/30 Read block 470 len 28672 12/30 Write block 56 len 16384 data: 3 3 3 3 13/30 Write block 582 len 4096 data: 2 14/30 Read block 295 len 36864 15/30 Flush 16/30 Write block 100 len 4096 data: 2 17/30 Read block 488 len 32768 18/30 Read block 110 len 16384 19/30 Write block 206 len 16384 data: 3 2 2 2 20/30 Read block 72 len 20480 21/30 Read block 94 len 12288 22/30 Write block 309 len 36864 data: 5 6 6 5 5 5 4 4 5 23/30 Read block 98 len 12288 24/30 Read block 472 len 36864 25/30 Write block 543 len 12288 data: 2 2 3 26/30 Write block 267 len 28672 data: 4 4 4 3 2 2 2 27/30 Read block 242 len 40960 28/30 Read block 421 len 28672 29/30 Read block 474 len 4096 30/30 Write block 456 len 4096 data: 2 ---------------------------------------------------------------- Crucible gen:14 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 3 Done Done Done false 1002 Acked Write 2 Done Done Done false 1003 Acked Read 2 Done Done Done false 1004 Acked Read 4 Done Done Done false 1005 Acked Read 1 Done Done Done false 1006 Acked Read 10 Done Done Done false 1007 Acked Write 3 Done Done Done false 1008 Acked Write 4 Done Done Done false 1009 Acked Read 7 Done Done Done false 1010 Acked Read 7 Done Sent Done false 1011 Acked Write 4 Done Sent Done false 1012 Acked Write 1 Done Sent Done false 1013 Acked Read 9 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Read 4 Done Sent Done false 1018 Acked Write 4 Done Sent Done false 1019 Acked Read 5 Done Sent Done false 1020 Acked Read 3 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Read 3 Done Sent Done false 1023 Acked Read 9 Done Sent Done false 1024 Acked Write 3 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Read 10 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 20 1 22 Done 28 9 28 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1000)) Some(JobId(1014)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:47.583634288Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:47.583817684Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1189,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 26 26 26 F F F 1 020-039 12 12 12 25 25 25 F F F 2 040-059 14 6 14 28 12 28 F F F <--- 3 060-079 13 13 13 26 26 26 F F F 4 080-099 14 10 14 28 19 28 F T F <--- 5 100-119 6 6 6 12 12 12 T F T <--- 6 120-139 12 12 12 23 23 23 F F F 7 140-159 12 12 12 25 25 25 F F F 8 160-179 6 6 6 12 12 12 F F F 9 180-199 14 12 14 28 25 28 F T F <--- 10 200-219 10 10 10 19 19 19 T F T <--- 11 220-239 10 10 10 19 19 19 F F F 12 240-259 5 5 5 10 10 10 F F F 13 260-279 12 12 12 25 25 25 T F T <--- 14 280-299 4 4 4 9 9 9 F F F 15 300-319 12 12 12 25 25 25 T F T <--- 16 320-339 12 12 12 25 25 25 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 10 10 10 19 19 19 F F F 19 380-399 8 8 8 15 15 15 F F F 20 400-419 8 8 8 15 15 15 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 10 10 10 19 19 19 T F T <--- 23 460-479 1 1 1 1 1 1 F F F 24 480-499 9 9 9 18 18 18 F F F 25 500-519 12 12 12 23 23 23 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 9 9 9 18 18 18 T F T <--- 28 560-579 14 11 14 28 20 28 F T F <--- 29 580-599 14 4 14 28 9 28 F T F <--- Max gen: 14, Max flush: 28 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 7 Verifying data now No change after verify Loop: 7 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 26 26 26 F F F 1 020-039 12 12 12 25 25 25 F F F 2 040-059 14 14 14 28 28 28 F F F 3 060-079 13 13 13 26 26 26 F F F 4 080-099 14 14 14 28 28 28 F F F 5 100-119 15 15 15 29 29 29 F F F 6 120-139 12 12 12 23 23 23 F F F 7 140-159 12 12 12 25 25 25 F F F 8 160-179 6 6 6 12 12 12 F F F 9 180-199 14 14 14 28 28 28 F F F 10 200-219 15 15 15 29 29 29 F F F 11 220-239 10 10 10 19 19 19 F F F 12 240-259 5 5 5 10 10 10 F F F 13 260-279 15 15 15 29 29 29 F F F 14 280-299 4 4 4 9 9 9 F F F 15 300-319 15 15 15 29 29 29 F F F 16 320-339 12 12 12 25 25 25 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 10 10 10 19 19 19 F F F 19 380-399 8 8 8 15 15 15 F F F 20 400-419 8 8 8 15 15 15 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 15 15 15 29 29 29 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 9 9 9 18 18 18 F F F 25 500-519 12 12 12 23 23 23 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 15 15 15 29 29 29 F F F 28 560-579 14 14 14 28 28 28 F F F 29 580-599 14 14 14 28 28 28 F F F Max gen: 15, Max flush: 29 Begin loop 8 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.829974355Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:51.837173017Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.837233545Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.844338605Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.844637649Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.844671912Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.8446812Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.844695718Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197} {"msg":"Crucible 3821992c-2f52-4b5c-91a6-424b33ca0199 has session id: f472813d-0ad8-41f3-b0de-90c88c0418aa","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.844978035Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"Upstairs opts: Upstairs UUID: 3821992c-2f52-4b5c-91a6-424b33ca0199, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.84500148Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"Crucible stats registered with UUID: 3821992c-2f52-4b5c-91a6-424b33ca0199","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845011559Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:51.845021997Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845180527Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"io task","client":"2","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845196864Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"io task","client":"0","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845206862Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"io task","client":"1","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845371081Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"task":"crutest"} {"msg":"The guest has requested activation with gen:16","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845387558Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197} {"msg":"3821992c-2f52-4b5c-91a6-424b33ca0199 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845420402Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.84542978Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"client":"0","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845439318Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"client":"1","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845448767Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"client":"2","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845458325Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"io task","client":"2","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845469603Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"io task","client":"0","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845479471Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"io task","client":"1","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"ds_connection connected from Ok(127.0.0.1:39870)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.84554001Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"io task","client":"0","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"ds_connection connected from Ok(127.0.0.1:58760)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845550898Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"io task","client":"2","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"ds_connection connected from Ok(127.0.0.1:39318)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.845561096Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"io task","client":"1","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.84628705Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"client":"1","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846328212Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"client":"2","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.84634039Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"client":"0","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846608Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846630645Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"[0]R flush_numbers[0..12]: [26, 25, 28, 26, 28, 29, 23, 25, 12, 28, 29, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846685745Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"downstairs","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"[0]R generation[0..12]: [13, 12, 14, 13, 14, 15, 12, 12, 6, 14, 15, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846697093Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"downstairs","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846708101Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"downstairs","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"[1]R flush_numbers[0..12]: [26, 25, 28, 26, 28, 29, 23, 25, 12, 28, 29, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846718489Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"downstairs","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"[1]R generation[0..12]: [13, 12, 14, 13, 14, 15, 12, 12, 6, 14, 15, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846728717Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"downstairs","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846741635Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"downstairs","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"[2]R flush_numbers[0..12]: [26, 25, 28, 26, 28, 29, 23, 25, 12, 28, 29, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846752013Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"downstairs","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"[2]R generation[0..12]: [13, 12, 14, 13, 14, 15, 12, 12, 6, 14, 15, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846762271Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"downstairs","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846773029Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"downstairs","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"Max found gen is 16","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846782307Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"downstairs","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"Generation requested: 16 >= found:16","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846791925Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"downstairs","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"Next flush: 30","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846801073Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"downstairs","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846810382Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"":"downstairs","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.84682003Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846828888Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"client":"0","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846838566Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"client":"1","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846848065Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"client":"2","":"downstairs","session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846857913Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846866971Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"3821992c-2f52-4b5c-91a6-424b33ca0199 is now active with session: f472813d-0ad8-41f3-b0de-90c88c0418aa","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846876729Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846886927Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"session_id":"f472813d-0ad8-41f3-b0de-90c88c0418aa"} {"msg":"The guest has finished waiting for activation with:16","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846895826Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197} {"msg":"Activated sub_volume 3821992c-2f52-4b5c-91a6-424b33ca0199","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846904044Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.846912123Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.84724637Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"task":"crutest"} 01/30 Write block 91 len 24576 data: 3 3 2 3 3 3 02/30 Write block 381 len 24576 data: 6 6 6 5 3 3 03/30 Read block 571 len 24576 04/30 Read block 545 len 16384 05/30 Read block 531 len 32768 06/30 Flush 07/30 Write block 103 len 40960 data: 2 2 2 2 2 2 2 2 2 3 08/30 Read block 477 len 40960 09/30 Read block 40 len 36864 10/30 Read block 338 len 8192 11/30 Read block 368 len 24576 12/30 Read block 277 len 36864 13/30 Write block 326 len 28672 data: 2 2 3 3 3 2 3 14/30 Read block 591 len 16384 15/30 Write block 207 len 16384 data: 3 3 3 2 16/30 Read block 525 len 36864 17/30 Write block 204 len 4096 data: 2 18/30 Write block 196 len 8192 data: 3 2 19/30 Write block 84 len 32768 data: 4 4 4 3 3 3 3 4 20/30 Write block 195 len 12288 data: 3 4 3 21/30 Write block 404 len 28672 data: 2 2 2 2 2 2 2 22/30 Read block 438 len 32768 23/30 Read block 6 len 8192 24/30 Read block 495 len 4096 25/30 Read block 172 len 28672 26/30 Read block 50 len 12288 27/30 Read block 286 len 40960 28/30 Read block 54 len 40960 29/30 Read block 39 len 8192 30/30 Write block 125 len 36864 data: 5 5 4 4 4 5 4 4 4 ---------------------------------------------------------------- Crucible gen:16 GIO:true work queues: Upstairs:0 downstairs:24 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1006 Acked Write 10 Done Done Done false 1007 Acked Read 10 Done Done Done false 1008 Acked Read 9 Done Done Done false 1009 Acked Read 2 Done Done Done false 1010 Acked Read 6 Done Done Done false 1011 Acked Read 9 Done Done Done false 1012 Acked Write 7 Done Done Done false 1013 Acked Read 4 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Write 2 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Write 3 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Read 8 Sent Done Done false 1022 Acked Read 2 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Read 7 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 17 1 1 19 Done 7 23 23 53 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1005)) Some(JobId(1005)) Some(JobId(1005)) Downstairs last five completed: 1005 1004 1003 1002 1001 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:51.856306883Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:51.856474362Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1197,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 26 26 26 F F F 1 020-039 12 12 12 25 25 25 F F F 2 040-059 14 14 14 28 28 28 F F F 3 060-079 13 13 13 26 26 26 F F F 4 080-099 16 16 16 30 30 30 F T T <--- 5 100-119 15 15 15 29 29 29 T T T 6 120-139 12 12 12 23 23 23 F T T <--- 7 140-159 12 12 12 25 25 25 F F F 8 160-179 6 6 6 12 12 12 F F F 9 180-199 14 14 14 28 28 28 F T T <--- 10 200-219 15 15 15 29 29 29 F T T <--- 11 220-239 10 10 10 19 19 19 F F F 12 240-259 5 5 5 10 10 10 F F F 13 260-279 15 15 15 29 29 29 F F F 14 280-299 4 4 4 9 9 9 F F F 15 300-319 15 15 15 29 29 29 F F F 16 320-339 12 12 12 25 25 25 T T T 17 340-359 1 1 1 1 1 1 F F F 18 360-379 10 10 10 19 19 19 F F F 19 380-399 16 16 16 30 30 30 F F F 20 400-419 8 8 8 15 15 15 F T T <--- 21 420-439 4 4 4 9 9 9 F F F 22 440-459 15 15 15 29 29 29 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 9 9 9 18 18 18 F F F 25 500-519 12 12 12 23 23 23 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 15 15 15 29 29 29 F F F 28 560-579 14 14 14 28 28 28 F F F 29 580-599 14 14 14 28 28 28 F F F Max gen: 16, Max flush: 30 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 8 Verifying data now No change after verify Loop: 8 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 26 26 26 F F F 1 020-039 12 12 12 25 25 25 F F F 2 040-059 14 14 14 28 28 28 F F F 3 060-079 13 13 13 26 26 26 F F F 4 080-099 17 17 17 31 31 31 F F F 5 100-119 17 17 17 31 31 31 F F F 6 120-139 17 17 17 31 31 31 F F F 7 140-159 12 12 12 25 25 25 F F F 8 160-179 6 6 6 12 12 12 F F F 9 180-199 17 17 17 31 31 31 F F F 10 200-219 17 17 17 31 31 31 F F F 11 220-239 10 10 10 19 19 19 F F F 12 240-259 5 5 5 10 10 10 F F F 13 260-279 15 15 15 29 29 29 F F F 14 280-299 4 4 4 9 9 9 F F F 15 300-319 15 15 15 29 29 29 F F F 16 320-339 17 17 17 31 31 31 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 10 10 10 19 19 19 F F F 19 380-399 16 16 16 30 30 30 F F F 20 400-419 17 17 17 31 31 31 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 15 15 15 29 29 29 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 9 9 9 18 18 18 F F F 25 500-519 12 12 12 23 23 23 F F F 26 520-539 1 1 1 1 1 1 F F F 27 540-559 15 15 15 29 29 29 F F F 28 560-579 14 14 14 28 28 28 F F F 29 580-599 14 14 14 28 28 28 F F F Max gen: 17, Max flush: 31 Begin loop 9 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.097244592Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.105336007Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.1056374Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.105666155Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.105674903Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.105686251Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205} {"msg":"Crucible 0b293857-35b7-4762-82ca-6419ae225029 has session id: 61142cf9-fab7-4309-8558-b786adf9c57f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106027867Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"Upstairs opts: Upstairs UUID: 0b293857-35b7-4762-82ca-6419ae225029, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106049243Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"Crucible stats registered with UUID: 0b293857-35b7-4762-82ca-6419ae225029","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106058831Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:56.106068059Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106215182Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106248246Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106259304Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106396268Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"task":"crutest"} {"msg":"The guest has requested activation with gen:18","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106411115Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205} {"msg":"0b293857-35b7-4762-82ca-6419ae225029 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106444699Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106454187Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"client":"0","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106463895Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"client":"1","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106473584Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"client":"2","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106483112Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.10649299Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106502918Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"ds_connection connected from Ok(127.0.0.1:36578)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106551789Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"io task","client":"2","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"ds_connection connected from Ok(127.0.0.1:61209)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106562597Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"io task","client":"0","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"ds_connection connected from Ok(127.0.0.1:33046)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.106577664Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"io task","client":"1","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107244409Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"client":"0","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107270204Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"client":"1","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107334732Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"client":"2","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107543463Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.10756072Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"[0]R flush_numbers[0..12]: [26, 25, 28, 26, 31, 31, 31, 25, 12, 31, 31, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107617329Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"downstairs","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"[0]R generation[0..12]: [13, 12, 14, 13, 17, 17, 17, 12, 6, 17, 17, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107628567Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"downstairs","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107639465Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"downstairs","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"[1]R flush_numbers[0..12]: [26, 25, 28, 26, 31, 31, 31, 25, 12, 31, 31, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107649733Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"downstairs","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"[1]R generation[0..12]: [13, 12, 14, 13, 17, 17, 17, 12, 6, 17, 17, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107660001Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"downstairs","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107670709Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"downstairs","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"[2]R flush_numbers[0..12]: [26, 25, 28, 26, 31, 31, 31, 25, 12, 31, 31, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107681007Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"downstairs","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"[2]R generation[0..12]: [13, 12, 14, 13, 17, 17, 17, 12, 6, 17, 17, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107691305Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"downstairs","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107702153Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"downstairs","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"Max found gen is 18","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107711452Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"downstairs","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"Generation requested: 18 >= found:18","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.10772114Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"downstairs","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"Next flush: 32","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107732178Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"downstairs","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107741516Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"":"downstairs","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107751224Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107759933Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"client":"0","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107769351Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"client":"1","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107778889Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"client":"2","":"downstairs","session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107788697Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107797676Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"0b293857-35b7-4762-82ca-6419ae225029 is now active with session: 61142cf9-fab7-4309-8558-b786adf9c57f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107807534Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107816182Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"session_id":"61142cf9-fab7-4309-8558-b786adf9c57f"} {"msg":"The guest has finished waiting for activation with:18","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.10782533Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205} {"msg":"Activated sub_volume 0b293857-35b7-4762-82ca-6419ae225029","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107833269Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.107841277Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.108156848Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"task":"crutest"} 01/30 Write block 85 len 8192 data: 5 5 02/30 Read block 199 len 12288 03/30 Read block 236 len 28672 04/30 Write block 482 len 24576 data: 3 3 3 3 4 4 05/30 Read block 528 len 4096 06/30 Read block 311 len 36864 07/30 Read block 563 len 32768 08/30 Read block 500 len 8192 09/30 Read block 228 len 8192 10/30 Write block 402 len 12288 data: 3 3 3 11/30 Write block 324 len 16384 data: 3 2 3 3 12/30 Read block 143 len 24576 13/30 Read block 387 len 24576 14/30 Read block 132 len 28672 15/30 Flush 16/30 Write block 299 len 40960 data: 2 2 2 2 3 4 4 6 6 5 17/30 Write block 524 len 16384 data: 2 2 2 2 18/30 Write block 123 len 36864 data: 5 6 6 6 5 5 5 6 5 19/30 Flush 20/30 Write block 193 len 28672 data: 5 4 4 5 4 2 2 21/30 Write block 54 len 28672 data: 2 2 4 4 4 4 4 22/30 Flush 23/30 Write block 241 len 36864 data: 2 2 2 2 2 2 2 2 2 24/30 Read block 490 len 16384 25/30 Write block 212 len 20480 data: 2 2 2 2 2 26/30 Read block 394 len 28672 27/30 Write block 172 len 16384 data: 3 3 3 2 28/30 Read block 282 len 28672 29/30 Write block 253 len 36864 data: 2 2 3 4 4 4 4 3 3 30/30 Write block 370 len 36864 data: 2 3 3 3 4 5 5 5 5 ---------------------------------------------------------------- Crucible gen:18 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Sent Done false 1001 Acked Read 3 Done Sent Done false 1002 Acked Read 7 Done Sent Done false 1003 Acked Write 6 Done Sent Done false 1004 Acked Read 1 Done Sent Done false 1005 Acked Read 9 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Read 2 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Read 6 Done Sent Done false 1012 Acked Read 6 Done Sent Done false 1013 Acked Read 7 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 9 Done Sent Done false 1023 Acked Read 4 Done Sent Done false 1024 Acked Write 5 Done Sent Done false 1025 Acked Read 7 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 30 2 34 Done 28 0 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:56:56.11695911Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:56:56.117140606Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1205,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 26 26 26 F F F 1 020-039 12 12 12 25 25 25 F F F 2 040-059 18 14 18 34 28 34 F F F <--- 3 060-079 18 13 18 34 26 34 F F F <--- 4 080-099 18 17 18 32 31 32 F F F <--- 5 100-119 17 17 17 31 31 31 F F F 6 120-139 18 17 18 33 31 33 F F F <--- 7 140-159 12 12 12 25 25 25 F F F 8 160-179 6 6 6 12 12 12 T F T <--- 9 180-199 18 17 18 34 31 34 F F F <--- 10 200-219 17 17 17 31 31 31 T F T <--- 11 220-239 10 10 10 19 19 19 F F F 12 240-259 5 5 5 10 10 10 T F T <--- 13 260-279 15 15 15 29 29 29 T F T <--- 14 280-299 18 4 18 33 9 33 F F F <--- 15 300-319 18 15 18 33 29 33 F F F <--- 16 320-339 18 17 18 32 31 32 F F F <--- 17 340-359 1 1 1 1 1 1 F F F 18 360-379 10 10 10 19 19 19 T F T <--- 19 380-399 16 16 16 30 30 30 F F F 20 400-419 18 17 18 32 31 32 F F F <--- 21 420-439 4 4 4 9 9 9 F F F 22 440-459 15 15 15 29 29 29 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 18 9 18 32 18 32 F F F <--- 25 500-519 12 12 12 23 23 23 F F F 26 520-539 18 1 18 33 1 33 F F F <--- 27 540-559 15 15 15 29 29 29 F F F 28 560-579 14 14 14 28 28 28 F F F 29 580-599 14 14 14 28 28 28 F F F Max gen: 18, Max flush: 34 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 9 Verifying data now No change after verify Loop: 9 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 26 26 26 F F F 1 020-039 12 12 12 25 25 25 F F F 2 040-059 18 18 18 34 34 34 F F F 3 060-079 18 18 18 34 34 34 F F F 4 080-099 18 18 18 32 32 32 F F F 5 100-119 17 17 17 31 31 31 F F F 6 120-139 18 18 18 33 33 33 F F F 7 140-159 12 12 12 25 25 25 F F F 8 160-179 19 19 19 35 35 35 F F F 9 180-199 18 18 18 34 34 34 F F F 10 200-219 19 19 19 35 35 35 F F F 11 220-239 10 10 10 19 19 19 F F F 12 240-259 19 19 19 35 35 35 F F F 13 260-279 19 19 19 35 35 35 F F F 14 280-299 18 18 18 33 33 33 F F F 15 300-319 18 18 18 33 33 33 F F F 16 320-339 18 18 18 32 32 32 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 19 19 19 35 35 35 F F F 19 380-399 16 16 16 30 30 30 F F F 20 400-419 18 18 18 32 32 32 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 15 15 15 29 29 29 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 18 18 18 32 32 32 F F F 25 500-519 12 12 12 23 23 23 F F F 26 520-539 18 18 18 33 33 33 F F F 27 540-559 15 15 15 29 29 29 F F F 28 560-579 14 14 14 28 28 28 F F F 29 580-599 14 14 14 28 28 28 F F F Max gen: 19, Max flush: 35 Begin loop 10 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.37278973Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.380790281Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381093114Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381123509Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381132497Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381144065Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213} {"msg":"Crucible d8682508-b52b-4a5a-ac5f-8e437fd03a09 has session id: 2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381506387Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"Upstairs opts: Upstairs UUID: d8682508-b52b-4a5a-ac5f-8e437fd03a09, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381534482Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"Crucible stats registered with UUID: d8682508-b52b-4a5a-ac5f-8e437fd03a09","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.38154574Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:00.381555188Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381698741Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"io task","client":"2","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381725706Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"io task","client":"0","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381736464Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"io task","client":"1","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381848963Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"task":"crutest"} {"msg":"The guest has requested activation with gen:20","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381861081Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213} {"msg":"d8682508-b52b-4a5a-ac5f-8e437fd03a09 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381879157Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381887836Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"client":"0","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381899443Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"client":"1","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381909042Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"client":"2","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.38191842Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"io task","client":"2","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381928348Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"io task","client":"0","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.381938206Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"io task","client":"1","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"ds_connection connected from Ok(127.0.0.1:42336)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.382001774Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"io task","client":"0","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"ds_connection connected from Ok(127.0.0.1:43451)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.38202727Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"io task","client":"2","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"ds_connection connected from Ok(127.0.0.1:51852)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.382083649Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"io task","client":"1","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.382645454Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"client":"0","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.3826631Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"client":"1","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.382727498Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"client":"2","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.382942988Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.382973922Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"[0]R flush_numbers[0..12]: [26, 25, 34, 34, 32, 31, 33, 25, 35, 34, 35, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383045149Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"downstairs","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"[0]R generation[0..12]: [13, 12, 18, 18, 18, 17, 18, 12, 19, 18, 19, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383056297Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"downstairs","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383069814Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"downstairs","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"[1]R flush_numbers[0..12]: [26, 25, 34, 34, 32, 31, 33, 25, 35, 34, 35, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383080222Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"downstairs","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"[1]R generation[0..12]: [13, 12, 18, 18, 18, 17, 18, 12, 19, 18, 19, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.3830905Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"downstairs","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383101348Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"downstairs","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"[2]R flush_numbers[0..12]: [26, 25, 34, 34, 32, 31, 33, 25, 35, 34, 35, 19]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383111756Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"downstairs","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"[2]R generation[0..12]: [13, 12, 18, 18, 18, 17, 18, 12, 19, 18, 19, 10]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383122034Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"downstairs","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383132982Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"downstairs","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"Max found gen is 20","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383142381Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"downstairs","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"Generation requested: 20 >= found:20","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383151959Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"downstairs","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"Next flush: 36","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383161207Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"downstairs","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383170685Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"":"downstairs","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383180364Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383189252Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"client":"0","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.38319891Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"client":"1","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383208658Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"client":"2","":"downstairs","session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383219826Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383228944Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"d8682508-b52b-4a5a-ac5f-8e437fd03a09 is now active with session: 2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383238763Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383247351Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"session_id":"2994f7f6-f1e3-4dad-9cc0-5b0f0f7eb79b"} {"msg":"The guest has finished waiting for activation with:20","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383256529Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213} {"msg":"Activated sub_volume d8682508-b52b-4a5a-ac5f-8e437fd03a09","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383264438Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383272446Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.383563262Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"task":"crutest"} 01/30 Write block 36 len 40960 data: 2 2 2 2 2 3 3 3 3 3 02/30 Read block 345 len 28672 03/30 Write block 274 len 16384 data: 2 2 2 2 04/30 Write block 499 len 36864 data: 2 2 2 2 3 3 4 3 3 05/30 Write block 123 len 32768 data: 6 7 7 7 6 6 6 7 06/30 Write block 378 len 12288 data: 6 5 6 07/30 Write block 65 len 20480 data: 4 4 4 4 3 08/30 Read block 397 len 20480 09/30 Write block 113 len 24576 data: 4 4 5 5 4 4 10/30 Write block 175 len 24576 data: 3 2 2 4 4 4 11/30 Read block 308 len 16384 12/30 Read block 360 len 24576 13/30 Read block 148 len 24576 14/30 Write block 111 len 12288 data: 3 4 5 15/30 Flush 16/30 Write block 205 len 40960 data: 2 4 4 4 4 3 2 3 3 3 17/30 Read block 257 len 24576 18/30 Flush 19/30 Write block 230 len 20480 data: 2 2 2 2 2 20/30 Flush 21/30 Write block 158 len 24576 data: 2 2 2 2 2 2 22/30 Read block 176 len 32768 23/30 Write block 124 len 16384 data: 8 8 8 7 24/30 Write block 186 len 40960 data: 4 3 2 4 4 5 5 6 5 5 25/30 Write block 516 len 40960 data: 3 3 3 2 2 2 2 2 3 3 26/30 Write block 284 len 28672 data: 3 3 3 3 3 3 3 27/30 Read block 140 len 40960 28/30 Read block 507 len 20480 29/30 Read block 50 len 28672 30/30 Read block 582 len 24576 ---------------------------------------------------------------- Crucible gen:20 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Write 4 Done Done Sent false 1003 Acked Write 9 Done Done Sent false 1004 Acked Write 8 Done Done Sent false 1005 Acked Write 3 Done Done Sent false 1006 Acked Write 5 Done Done Sent false 1007 Acked Read 5 Done Done Sent false 1008 Acked Write 6 Done Done Sent false 1009 Acked Write 6 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Read 6 Done Done Sent false 1012 Acked Read 6 Done Done Sent false 1013 Acked Write 3 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 5 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 6 Done Done Sent false 1021 Acked Read 8 Done Done Sent false 1022 Acked Write 4 Done Done Sent false 1023 Acked Write 10 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Read 10 Done Done Sent false 1027 Acked Read 5 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Read 6 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 28 29 Done 30 29 2 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:00.392345687Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:00.392530722Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1213,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 26 26 26 F F F 1 020-039 20 20 12 36 36 25 F F T <--- 2 040-059 20 20 18 36 36 34 F F T <--- 3 060-079 20 20 18 36 36 34 F F F <--- 4 080-099 18 18 18 32 32 32 F F F 5 100-119 20 20 17 36 36 31 F F F <--- 6 120-139 20 20 18 36 36 33 T T F <--- 7 140-159 12 12 12 25 25 25 T T F <--- 8 160-179 20 20 19 36 36 35 T T F <--- 9 180-199 20 20 18 36 36 34 T T F <--- 10 200-219 20 20 19 37 37 35 F F F <--- 11 220-239 20 20 10 38 38 19 F F F <--- 12 240-259 19 19 19 35 35 35 F F F 13 260-279 20 20 19 36 36 35 F F F <--- 14 280-299 18 18 18 33 33 33 T T F <--- 15 300-319 18 18 18 33 33 33 F F F 16 320-339 18 18 18 32 32 32 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 20 20 19 36 36 35 F F F <--- 19 380-399 20 20 16 36 36 30 F F F <--- 20 400-419 18 18 18 32 32 32 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 15 15 15 29 29 29 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 20 20 18 36 36 32 F F F <--- 25 500-519 20 20 12 36 36 23 T T F <--- 26 520-539 18 18 18 33 33 33 T T F <--- 27 540-559 15 15 15 29 29 29 F F F 28 560-579 14 14 14 28 28 28 F F F 29 580-599 14 14 14 28 28 28 F F F Max gen: 20, Max flush: 38 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 10 Verifying data now No change after verify Loop: 10 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 26 26 26 F F F 1 020-039 20 20 20 36 36 36 F F F 2 040-059 20 20 20 36 36 36 F F F 3 060-079 20 20 20 36 36 36 F F F 4 080-099 18 18 18 32 32 32 F F F 5 100-119 20 20 20 36 36 36 F F F 6 120-139 21 21 21 39 39 39 F F F 7 140-159 21 21 21 39 39 39 F F F 8 160-179 21 21 21 39 39 39 F F F 9 180-199 21 21 21 39 39 39 F F F 10 200-219 20 20 20 37 37 37 F F F 11 220-239 20 20 20 38 38 38 F F F 12 240-259 19 19 19 35 35 35 F F F 13 260-279 20 20 20 36 36 36 F F F 14 280-299 21 21 21 39 39 39 F F F 15 300-319 18 18 18 33 33 33 F F F 16 320-339 18 18 18 32 32 32 F F F 17 340-359 1 1 1 1 1 1 F F F 18 360-379 20 20 20 36 36 36 F F F 19 380-399 20 20 20 36 36 36 F F F 20 400-419 18 18 18 32 32 32 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 15 15 15 29 29 29 F F F 23 460-479 1 1 1 1 1 1 F F F 24 480-499 20 20 20 36 36 36 F F F 25 500-519 21 21 21 39 39 39 F F F 26 520-539 21 21 21 39 39 39 F F F 27 540-559 15 15 15 29 29 29 F F F 28 560-579 14 14 14 28 28 28 F F F 29 580-599 14 14 14 28 28 28 F F F Max gen: 21, Max flush: 39 Begin loop 11 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.644994453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:04.652170129Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.652224628Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.659132685Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.659395845Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.65942218Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.659430879Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.659444556Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221} {"msg":"Crucible 79642818-c87a-494b-bfe3-5b21e9b83521 has session id: a57272a5-8fbf-4094-8f7e-cf196ba5d8d9","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.659810858Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"Upstairs opts: Upstairs UUID: 79642818-c87a-494b-bfe3-5b21e9b83521, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.659831174Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"Crucible stats registered with UUID: 79642818-c87a-494b-bfe3-5b21e9b83521","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.659840992Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:04.65985Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660001412Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"io task","client":"0","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660017649Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"io task","client":"1","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660027627Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"io task","client":"2","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660194756Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"task":"crutest"} {"msg":"The guest has requested activation with gen:22","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660211233Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221} {"msg":"79642818-c87a-494b-bfe3-5b21e9b83521 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660243297Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660252585Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"client":"0","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660262113Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"client":"1","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660271641Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"client":"2","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.6602811Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"io task","client":"2","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660291338Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"io task","client":"0","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660302486Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"io task","client":"1","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"ds_connection connected from Ok(127.0.0.1:37084)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660364224Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"io task","client":"2","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"ds_connection connected from Ok(127.0.0.1:52161)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660389129Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"io task","client":"1","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"ds_connection connected from Ok(127.0.0.1:40095)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.660450978Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"io task","client":"0","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661041127Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"client":"1","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661053185Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"client":"2","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661063923Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"client":"0","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661374025Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661386413Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"[0]R flush_numbers[0..12]: [26, 36, 36, 36, 32, 36, 39, 39, 39, 39, 37, 38]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.66140284Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"downstairs","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"[0]R generation[0..12]: [13, 20, 20, 20, 18, 20, 21, 21, 21, 21, 20, 20]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661413518Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"downstairs","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661429675Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"downstairs","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"[1]R flush_numbers[0..12]: [26, 36, 36, 36, 32, 36, 39, 39, 39, 39, 37, 38]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661439783Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"downstairs","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"[1]R generation[0..12]: [13, 20, 20, 20, 18, 20, 21, 21, 21, 21, 20, 20]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661450021Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"downstairs","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661463188Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"downstairs","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"[2]R flush_numbers[0..12]: [26, 36, 36, 36, 32, 36, 39, 39, 39, 39, 37, 38]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661473496Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"downstairs","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"[2]R generation[0..12]: [13, 20, 20, 20, 18, 20, 21, 21, 21, 21, 20, 20]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661483804Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"downstairs","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661494822Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"downstairs","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"Max found gen is 22","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661504221Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"downstairs","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"Generation requested: 22 >= found:22","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661513769Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"downstairs","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"Next flush: 40","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661522937Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"downstairs","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661532275Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"":"downstairs","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661541903Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661550562Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"client":"0","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.66156013Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"client":"1","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661569648Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"client":"2","":"downstairs","session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661579456Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661588765Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"79642818-c87a-494b-bfe3-5b21e9b83521 is now active with session: a57272a5-8fbf-4094-8f7e-cf196ba5d8d9","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661598693Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661609331Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"session_id":"a57272a5-8fbf-4094-8f7e-cf196ba5d8d9"} {"msg":"The guest has finished waiting for activation with:22","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661618189Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221} {"msg":"Activated sub_volume 79642818-c87a-494b-bfe3-5b21e9b83521","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661626408Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661634646Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.661947438Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"task":"crutest"} 01/30 Write block 219 len 40960 data: 3 3 3 3 2 2 2 2 3 3 02/30 Write block 336 len 36864 data: 3 3 2 2 2 2 2 2 2 03/30 Write block 579 len 4096 data: 2 04/30 Read block 558 len 16384 05/30 Read block 266 len 28672 06/30 Read block 217 len 28672 07/30 Flush 08/30 Flush 09/30 Write block 477 len 28672 data: 2 2 2 2 2 4 4 10/30 Write block 586 len 40960 data: 2 3 3 3 3 3 3 3 3 3 11/30 Flush 12/30 Write block 141 len 28672 data: 3 2 2 3 3 2 3 13/30 Read block 190 len 12288 14/30 Write block 78 len 20480 data: 3 3 2 2 2 15/30 Read block 12 len 4096 16/30 Read block 280 len 12288 17/30 Write block 565 len 36864 data: 4 4 3 2 2 2 2 3 3 18/30 Read block 530 len 24576 19/30 Read block 95 len 28672 20/30 Read block 482 len 24576 21/30 Read block 497 len 24576 22/30 Write block 550 len 32768 data: 3 3 4 4 3 3 3 4 23/30 Write block 171 len 32768 data: 2 4 4 4 4 3 3 5 24/30 Flush 25/30 Write block 291 len 24576 data: 3 2 2 2 2 2 26/30 Read block 34 len 36864 27/30 Read block 380 len 4096 28/30 Write block 229 len 32768 data: 2 3 3 3 3 3 2 2 29/30 Read block 125 len 20480 30/30 Write block 374 len 16384 data: 5 6 6 6 ---------------------------------------------------------------- Crucible gen:22 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 9 Sent Done Done false 1002 Acked Write 1 Sent Done Done false 1003 Acked Read 4 Sent Done Done false 1004 Acked Read 7 Sent Done Done false 1005 Acked Read 7 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 7 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 7 Sent Done Done false 1012 Acked Read 3 Sent Done Done false 1013 Acked Write 5 Sent Done Done false 1014 Acked Read 1 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Read 6 Sent Done Done false 1018 Acked Read 7 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Write 8 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 8 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 1 31 Done 1 29 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:04.672003804Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:04.67218382Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1221,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 26 26 26 F F F 1 020-039 20 20 20 36 36 36 F F F 2 040-059 20 20 20 36 36 36 F F F 3 060-079 20 22 22 36 43 43 F F F <--- 4 080-099 18 22 22 32 43 43 F F F <--- 5 100-119 20 20 20 36 36 36 F F F 6 120-139 21 21 21 39 39 39 F F F 7 140-159 21 22 22 39 43 43 F F F <--- 8 160-179 21 22 22 39 43 43 F F F <--- 9 180-199 21 21 21 39 39 39 F F F 10 200-219 20 22 22 37 40 40 T F F <--- 11 220-239 20 22 22 38 40 40 T T T <--- 12 240-259 19 19 19 35 35 35 F F F 13 260-279 20 20 20 36 36 36 F F F 14 280-299 21 21 21 39 39 39 F T T <--- 15 300-319 18 18 18 33 33 33 F F F 16 320-339 18 22 22 32 40 40 F F F <--- 17 340-359 1 22 22 1 40 40 F F F <--- 18 360-379 20 20 20 36 36 36 F T T <--- 19 380-399 20 20 20 36 36 36 F F F 20 400-419 18 18 18 32 32 32 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 15 15 15 29 29 29 F F F 23 460-479 1 22 22 1 42 42 F F F <--- 24 480-499 20 22 22 36 42 42 F F F <--- 25 500-519 21 21 21 39 39 39 F F F 26 520-539 21 21 21 39 39 39 F F F 27 540-559 15 22 22 29 43 43 F F F <--- 28 560-579 14 22 22 28 43 43 F F F <--- 29 580-599 14 22 22 28 42 42 F F F <--- Max gen: 22, Max flush: 43 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 11 Verifying data now No change after verify Loop: 11 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 13 13 26 26 26 F F F 1 020-039 20 20 20 36 36 36 F F F 2 040-059 20 20 20 36 36 36 F F F 3 060-079 22 22 22 43 43 43 F F F 4 080-099 22 22 22 43 43 43 F F F 5 100-119 20 20 20 36 36 36 F F F 6 120-139 21 21 21 39 39 39 F F F 7 140-159 22 22 22 43 43 43 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 21 21 21 39 39 39 F F F 10 200-219 22 22 22 40 40 40 F F F 11 220-239 23 23 23 44 44 44 F F F 12 240-259 19 19 19 35 35 35 F F F 13 260-279 20 20 20 36 36 36 F F F 14 280-299 23 23 23 44 44 44 F F F 15 300-319 18 18 18 33 33 33 F F F 16 320-339 22 22 22 40 40 40 F F F 17 340-359 22 22 22 40 40 40 F F F 18 360-379 23 23 23 44 44 44 F F F 19 380-399 20 20 20 36 36 36 F F F 20 400-419 18 18 18 32 32 32 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 15 15 15 29 29 29 F F F 23 460-479 22 22 22 42 42 42 F F F 24 480-499 22 22 22 42 42 42 F F F 25 500-519 21 21 21 39 39 39 F F F 26 520-539 21 21 21 39 39 39 F F F 27 540-559 22 22 22 43 43 43 F F F 28 560-579 22 22 22 43 43 43 F F F 29 580-599 22 22 22 42 42 42 F F F Max gen: 23, Max flush: 44 Begin loop 12 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.928549739Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:08.935654529Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.935708659Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.94248389Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.94274986Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.942777195Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.942785813Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.942797301Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229} {"msg":"Crucible 00c10e58-217c-4d0d-a777-236e55183ddd has session id: 6b5332c6-eebc-4f27-b677-077ad1047b16","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943119811Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"Upstairs opts: Upstairs UUID: 00c10e58-217c-4d0d-a777-236e55183ddd, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943138177Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"Crucible stats registered with UUID: 00c10e58-217c-4d0d-a777-236e55183ddd","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943149825Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:08.943158743Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943300217Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"io task","client":"1","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943316514Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"io task","client":"0","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943326482Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"io task","client":"2","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943470535Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"task":"crutest"} {"msg":"The guest has requested activation with gen:24","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943483313Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229} {"msg":"00c10e58-217c-4d0d-a777-236e55183ddd active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.94349625Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943504979Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"client":"0","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943514477Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"client":"1","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943524025Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"client":"2","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943533473Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"io task","client":"2","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943543431Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"io task","client":"0","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.94355332Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"io task","client":"1","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"ds_connection connected from Ok(127.0.0.1:56641)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943629845Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"io task","client":"1","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"ds_connection connected from Ok(127.0.0.1:55990)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.94365518Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"io task","client":"0","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"ds_connection connected from Ok(127.0.0.1:49942)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.943668468Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"io task","client":"2","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944333743Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"client":"1","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944369697Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"client":"2","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944420567Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"client":"0","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944675439Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.94472305Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"[0]R flush_numbers[0..12]: [26, 36, 36, 43, 43, 36, 39, 43, 43, 39, 40, 44]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944771171Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"downstairs","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"[0]R generation[0..12]: [13, 20, 20, 22, 22, 20, 21, 22, 22, 21, 22, 23]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944782239Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"downstairs","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944793307Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"downstairs","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"[1]R flush_numbers[0..12]: [26, 36, 36, 43, 43, 36, 39, 43, 43, 39, 40, 44]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944803725Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"downstairs","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"[1]R generation[0..12]: [13, 20, 20, 22, 22, 20, 21, 22, 22, 21, 22, 23]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944814173Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"downstairs","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944825101Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"downstairs","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"[2]R flush_numbers[0..12]: [26, 36, 36, 43, 43, 36, 39, 43, 43, 39, 40, 44]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944835459Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"downstairs","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"[2]R generation[0..12]: [13, 20, 20, 22, 22, 20, 21, 22, 22, 21, 22, 23]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944845777Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"downstairs","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944858265Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"downstairs","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"Max found gen is 24","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944867783Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"downstairs","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"Generation requested: 24 >= found:24","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944877482Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"downstairs","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"Next flush: 45","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.94488682Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"downstairs","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944896248Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"":"downstairs","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944906026Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944914665Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"client":"0","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944924353Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"client":"1","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944933951Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"client":"2","":"downstairs","session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944943639Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944952697Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"00c10e58-217c-4d0d-a777-236e55183ddd is now active with session: 6b5332c6-eebc-4f27-b677-077ad1047b16","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944962536Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"The guest has finished waiting for activation with:24","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944971674Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944979422Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"session_id":"6b5332c6-eebc-4f27-b677-077ad1047b16"} {"msg":"Activated sub_volume 00c10e58-217c-4d0d-a777-236e55183ddd","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944988541Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.944996669Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.945332586Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"task":"crutest"} 01/30 Write block 498 len 40960 data: 2 3 3 3 3 4 4 5 4 4 02/30 Read block 428 len 40960 03/30 Read block 171 len 24576 04/30 Read block 178 len 16384 05/30 Flush 06/30 Write block 396 len 12288 data: 3 3 3 07/30 Read block 350 len 20480 08/30 Read block 426 len 32768 09/30 Write block 236 len 8192 data: 3 2 10/30 Read block 27 len 40960 11/30 Write block 54 len 36864 data: 3 3 5 5 5 5 5 4 5 12/30 Read block 423 len 8192 13/30 Write block 559 len 32768 data: 3 3 3 3 3 3 5 5 14/30 Write block 80 len 12288 data: 3 3 3 15/30 Read block 346 len 40960 16/30 Write block 11 len 28672 data: 4 4 3 3 3 3 2 17/30 Write block 109 len 8192 data: 3 3 18/30 Read block 328 len 28672 19/30 Read block 201 len 40960 20/30 Flush 21/30 Write block 335 len 4096 data: 3 22/30 Read block 114 len 8192 23/30 Read block 399 len 40960 24/30 Read block 384 len 8192 25/30 Write block 252 len 16384 data: 2 3 3 4 26/30 Read block 482 len 32768 27/30 Read block 501 len 28672 28/30 Read block 346 len 28672 29/30 Read block 388 len 24576 30/30 Read block 113 len 24576 ---------------------------------------------------------------- Crucible gen:24 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 10 Done Done Done false 1002 Acked Read 6 Done Done Done false 1003 Acked Read 4 Done Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 3 Sent Done Done false 1006 Acked Read 5 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Read 10 Sent Done Done false 1010 Acked Write 9 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Write 8 Sent Done Done false 1013 Acked Write 3 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Write 2 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Read 10 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 1 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Read 10 Sent Done Done false 1023 Acked Read 2 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Read 8 Sent Done Done false 1026 Acked Read 7 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Read 6 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 0 0 26 Done 4 30 30 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:08.954938907Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:08.955122343Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1229,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 13 24 24 26 46 46 F F F <--- 1 020-039 20 20 20 36 36 36 F F F 2 040-059 20 24 24 36 46 46 F F F <--- 3 060-079 22 24 24 43 46 46 F F F <--- 4 080-099 22 24 24 43 46 46 F F F <--- 5 100-119 20 24 24 36 46 46 F F F <--- 6 120-139 21 21 21 39 39 39 F F F 7 140-159 22 22 22 43 43 43 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 21 21 21 39 39 39 F F F 10 200-219 22 22 22 40 40 40 F F F 11 220-239 23 24 24 44 46 46 F F F <--- 12 240-259 19 19 19 35 35 35 F T T <--- 13 260-279 20 20 20 36 36 36 F F F 14 280-299 23 23 23 44 44 44 F F F 15 300-319 18 18 18 33 33 33 F F F 16 320-339 22 22 22 40 40 40 F T T <--- 17 340-359 22 22 22 40 40 40 F F F 18 360-379 23 23 23 44 44 44 F F F 19 380-399 20 24 24 36 46 46 F F F <--- 20 400-419 18 18 18 32 32 32 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 15 15 15 29 29 29 F F F 23 460-479 22 22 22 42 42 42 F F F 24 480-499 22 24 24 42 45 45 T F F <--- 25 500-519 21 24 24 39 45 45 T F F <--- 26 520-539 21 21 21 39 39 39 F F F 27 540-559 22 24 24 43 46 46 F F F <--- 28 560-579 22 24 24 43 46 46 F F F <--- 29 580-599 22 22 22 42 42 42 F F F Max gen: 24, Max flush: 46 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 12 Verifying data now No change after verify Loop: 12 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 46 46 46 F F F 1 020-039 20 20 20 36 36 36 F F F 2 040-059 24 24 24 46 46 46 F F F 3 060-079 24 24 24 46 46 46 F F F 4 080-099 24 24 24 46 46 46 F F F 5 100-119 24 24 24 46 46 46 F F F 6 120-139 21 21 21 39 39 39 F F F 7 140-159 22 22 22 43 43 43 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 21 21 21 39 39 39 F F F 10 200-219 22 22 22 40 40 40 F F F 11 220-239 24 24 24 46 46 46 F F F 12 240-259 25 25 25 47 47 47 F F F 13 260-279 20 20 20 36 36 36 F F F 14 280-299 23 23 23 44 44 44 F F F 15 300-319 18 18 18 33 33 33 F F F 16 320-339 25 25 25 47 47 47 F F F 17 340-359 22 22 22 40 40 40 F F F 18 360-379 23 23 23 44 44 44 F F F 19 380-399 24 24 24 46 46 46 F F F 20 400-419 18 18 18 32 32 32 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 15 15 15 29 29 29 F F F 23 460-479 22 22 22 42 42 42 F F F 24 480-499 24 24 24 45 45 45 F F F 25 500-519 24 24 24 45 45 45 F F F 26 520-539 21 21 21 39 39 39 F F F 27 540-559 24 24 24 46 46 46 F F F 28 560-579 24 24 24 46 46 46 F F F 29 580-599 22 22 22 42 42 42 F F F Max gen: 25, Max flush: 47 Begin loop 13 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.202978856Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.2112346Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.211526385Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.2115549Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.211563388Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.211574796Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237} {"msg":"Crucible 954b4c75-13eb-45dc-becf-0585e84ad66f has session id: 1bd93941-d77a-44a6-a808-fc62ac51e0fd","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.211949316Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"Upstairs opts: Upstairs UUID: 954b4c75-13eb-45dc-becf-0585e84ad66f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.21197912Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"Crucible stats registered with UUID: 954b4c75-13eb-45dc-becf-0585e84ad66f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.211990308Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:13.212008365Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.212145459Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"io task","client":"1","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.212177123Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"io task","client":"0","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.212187851Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"io task","client":"2","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.212305249Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"task":"crutest"} {"msg":"The guest has requested activation with gen:26","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.212316897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237} {"msg":"954b4c75-13eb-45dc-becf-0585e84ad66f active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.212334564Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.212343302Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"client":"0","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.21235291Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"client":"1","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.212362658Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"client":"2","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.212372197Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"io task","client":"2","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.212382075Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"io task","client":"0","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.212391873Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"io task","client":"1","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"ds_connection connected from Ok(127.0.0.1:50242)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.21245873Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"io task","client":"2","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"ds_connection connected from Ok(127.0.0.1:45313)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.212483686Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"io task","client":"1","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"ds_connection connected from Ok(127.0.0.1:46439)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.212495244Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"io task","client":"0","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213181995Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"client":"1","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.2132062Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"client":"0","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213262Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"client":"2","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213482189Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213517082Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"[0]R flush_numbers[0..12]: [46, 36, 46, 46, 46, 46, 39, 43, 43, 39, 40, 46]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213607425Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"downstairs","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"[0]R generation[0..12]: [24, 20, 24, 24, 24, 24, 21, 22, 22, 21, 22, 24]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213618603Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"downstairs","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213629551Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"downstairs","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"[1]R flush_numbers[0..12]: [46, 36, 46, 46, 46, 46, 39, 43, 43, 39, 40, 46]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213639969Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"downstairs","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"[1]R generation[0..12]: [24, 20, 24, 24, 24, 24, 21, 22, 22, 21, 22, 24]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213650197Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"downstairs","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213660945Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"downstairs","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"[2]R flush_numbers[0..12]: [46, 36, 46, 46, 46, 46, 39, 43, 43, 39, 40, 46]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213671063Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"downstairs","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"[2]R generation[0..12]: [24, 20, 24, 24, 24, 24, 21, 22, 22, 21, 22, 24]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213681372Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"downstairs","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213692079Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"downstairs","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"Max found gen is 26","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213701428Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"downstairs","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"Generation requested: 26 >= found:26","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213711156Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"downstairs","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"Next flush: 48","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213720384Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"downstairs","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213729732Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"":"downstairs","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213739511Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213749399Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"client":"0","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213758977Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"client":"1","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213768545Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"client":"2","":"downstairs","session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213778293Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213787512Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"954b4c75-13eb-45dc-becf-0585e84ad66f is now active with session: 1bd93941-d77a-44a6-a808-fc62ac51e0fd","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.21379733Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213806058Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"session_id":"1bd93941-d77a-44a6-a808-fc62ac51e0fd"} {"msg":"The guest has finished waiting for activation with:26","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213814946Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237} {"msg":"Activated sub_volume 954b4c75-13eb-45dc-becf-0585e84ad66f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213822965Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.213830913Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.214151583Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"task":"crutest"} 01/30 Write block 515 len 12288 data: 3 4 4 02/30 Write block 151 len 36864 data: 2 2 2 2 2 2 2 3 3 03/30 Write block 307 len 32768 data: 7 6 6 7 7 6 6 6 04/30 Read block 48 len 20480 05/30 Write block 393 len 4096 data: 2 06/30 Write block 528 len 36864 data: 2 2 2 2 2 2 2 2 2 07/30 Write block 224 len 32768 data: 3 3 3 4 4 3 4 4 08/30 Write block 93 len 16384 data: 3 4 4 4 09/30 Flush 10/30 Write block 251 len 40960 data: 2 3 4 4 5 5 5 5 5 4 11/30 Read block 312 len 28672 12/30 Read block 381 len 12288 13/30 Write block 391 len 12288 data: 2 2 3 14/30 Read block 534 len 28672 15/30 Read block 28 len 40960 16/30 Read block 228 len 16384 17/30 Read block 525 len 20480 18/30 Read block 525 len 32768 19/30 Read block 386 len 28672 20/30 Read block 550 len 8192 21/30 Read block 105 len 24576 22/30 Write block 53 len 16384 data: 2 4 4 6 23/30 Read block 410 len 16384 24/30 Write block 356 len 20480 data: 2 2 2 2 2 25/30 Read block 16 len 28672 26/30 Read block 577 len 8192 27/30 Read block 211 len 4096 28/30 Read block 229 len 4096 29/30 Write block 32 len 16384 data: 3 3 3 3 30/30 Read block 411 len 16384 ---------------------------------------------------------------- Crucible gen:26 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 9 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Read 5 Done Done Done false 1004 Acked Write 1 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Write 8 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 10 Done Done Sent false 1010 Acked Read 7 Done Done Sent false 1011 Acked Read 3 Done Done Sent false 1012 Acked Write 3 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Read 10 Done Done Sent false 1015 Acked Read 4 Done Done Sent false 1016 Acked Read 5 Done Done Sent false 1017 Acked Read 8 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Read 2 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Write 4 Done Done Sent false 1022 Acked Read 4 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Read 7 Done Done Sent false 1025 Acked Read 2 Done Done Sent false 1026 Acked Read 1 Done Done Sent false 1027 Acked Read 1 Done Done Sent false 1028 Acked Write 4 Done Done Sent false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 26 27 Done 30 29 4 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1008)) Some(JobId(1008)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:13.222521476Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:13.222698223Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1237,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 46 46 46 F F F 1 020-039 20 20 20 36 36 36 T T F <--- 2 040-059 24 24 24 46 46 46 T T F <--- 3 060-079 24 24 24 46 46 46 F F F 4 080-099 26 26 24 48 48 46 F F F <--- 5 100-119 24 24 24 46 46 46 F F F 6 120-139 21 21 21 39 39 39 F F F 7 140-159 26 26 22 48 48 43 F F T <--- 8 160-179 22 22 22 43 43 43 F F F 9 180-199 21 21 21 39 39 39 F F F 10 200-219 22 22 22 40 40 40 F F F 11 220-239 26 26 24 48 48 46 F F F <--- 12 240-259 25 25 25 47 47 47 T T F <--- 13 260-279 20 20 20 36 36 36 T T F <--- 14 280-299 23 23 23 44 44 44 F F F 15 300-319 26 26 18 48 48 33 F F T <--- 16 320-339 25 25 25 47 47 47 F F F 17 340-359 22 22 22 40 40 40 T T F <--- 18 360-379 23 23 23 44 44 44 T T F <--- 19 380-399 26 26 24 48 48 46 T T F <--- 20 400-419 18 18 18 32 32 32 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 15 15 15 29 29 29 F F F 23 460-479 22 22 22 42 42 42 F F F 24 480-499 24 24 24 45 45 45 F F F 25 500-519 26 26 24 48 48 45 F F T <--- 26 520-539 26 26 21 48 48 39 F F F <--- 27 540-559 24 24 24 46 46 46 F F F 28 560-579 24 24 24 46 46 46 F F F 29 580-599 22 22 22 42 42 42 F F F Max gen: 26, Max flush: 48 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 13 Verifying data now No change after verify Loop: 13 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 46 46 46 F F F 1 020-039 27 27 27 49 49 49 F F F 2 040-059 27 27 27 49 49 49 F F F 3 060-079 24 24 24 46 46 46 F F F 4 080-099 26 26 26 48 48 48 F F F 5 100-119 24 24 24 46 46 46 F F F 6 120-139 21 21 21 39 39 39 F F F 7 140-159 26 26 26 48 48 48 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 21 21 21 39 39 39 F F F 10 200-219 22 22 22 40 40 40 F F F 11 220-239 26 26 26 48 48 48 F F F 12 240-259 27 27 27 49 49 49 F F F 13 260-279 27 27 27 49 49 49 F F F 14 280-299 23 23 23 44 44 44 F F F 15 300-319 26 26 26 48 48 48 F F F 16 320-339 25 25 25 47 47 47 F F F 17 340-359 27 27 27 49 49 49 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 27 27 27 49 49 49 F F F 20 400-419 18 18 18 32 32 32 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 15 15 15 29 29 29 F F F 23 460-479 22 22 22 42 42 42 F F F 24 480-499 24 24 24 45 45 45 F F F 25 500-519 26 26 26 48 48 48 F F F 26 520-539 26 26 26 48 48 48 F F F 27 540-559 24 24 24 46 46 46 F F F 28 560-579 24 24 24 46 46 46 F F F 29 580-599 22 22 22 42 42 42 F F F Max gen: 27, Max flush: 49 Begin loop 14 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.469542196Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.477640359Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.477933524Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.477968928Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.477978346Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.477989994Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245} {"msg":"Crucible ffef0d90-8470-491f-98c9-fc8d4928256c has session id: 9eb5d872-5d77-474a-8833-99ca87fa178d","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478340698Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"Upstairs opts: Upstairs UUID: ffef0d90-8470-491f-98c9-fc8d4928256c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478366933Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"Crucible stats registered with UUID: ffef0d90-8470-491f-98c9-fc8d4928256c","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478376881Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:17.47838574Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478525363Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"io task","client":"1","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.4785423Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"io task","client":"0","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478552688Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"io task","client":"2","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478715468Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"task":"crutest"} {"msg":"The guest has requested activation with gen:28","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478739623Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245} {"msg":"ffef0d90-8470-491f-98c9-fc8d4928256c active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.4787581Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478766768Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"client":"0","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478776416Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"client":"1","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478785805Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"client":"2","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478795203Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"io task","client":"2","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478805011Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"io task","client":"0","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478816719Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"io task","client":"1","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"ds_connection connected from Ok(127.0.0.1:54799)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478910221Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"io task","client":"1","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"ds_connection connected from Ok(127.0.0.1:64992)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478941925Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"io task","client":"0","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"ds_connection connected from Ok(127.0.0.1:46086)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.478985337Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"io task","client":"2","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.479706492Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"client":"1","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.479731368Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"client":"0","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.479743165Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"client":"2","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480017284Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480052357Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"[0]R flush_numbers[0..12]: [46, 49, 49, 46, 48, 46, 39, 48, 43, 39, 40, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480110586Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"downstairs","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"[0]R generation[0..12]: [24, 27, 27, 24, 26, 24, 21, 26, 22, 21, 22, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480121634Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"downstairs","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480132462Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"downstairs","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"[1]R flush_numbers[0..12]: [46, 49, 49, 46, 48, 46, 39, 48, 43, 39, 40, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.48014279Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"downstairs","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"[1]R generation[0..12]: [24, 27, 27, 24, 26, 24, 21, 26, 22, 21, 22, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480153228Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"downstairs","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480166286Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"downstairs","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"[2]R flush_numbers[0..12]: [46, 49, 49, 46, 48, 46, 39, 48, 43, 39, 40, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480176694Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"downstairs","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"[2]R generation[0..12]: [24, 27, 27, 24, 26, 24, 21, 26, 22, 21, 22, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480186922Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"downstairs","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.48019781Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"downstairs","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"Max found gen is 28","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480207238Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"downstairs","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"Generation requested: 28 >= found:28","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480216707Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"downstairs","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"Next flush: 50","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480226015Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"downstairs","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480235363Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"":"downstairs","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480244951Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.48025357Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"client":"0","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480263128Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"client":"1","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480272536Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"client":"2","":"downstairs","session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480282294Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480291313Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"ffef0d90-8470-491f-98c9-fc8d4928256c is now active with session: 9eb5d872-5d77-474a-8833-99ca87fa178d","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480301051Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480310779Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"session_id":"9eb5d872-5d77-474a-8833-99ca87fa178d"} {"msg":"The guest has finished waiting for activation with:28","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480319707Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245} {"msg":"Activated sub_volume ffef0d90-8470-491f-98c9-fc8d4928256c","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480327846Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480335884Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.480637348Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"task":"crutest"} 01/30 Write block 142 len 4096 data: 3 02/30 Read block 450 len 16384 03/30 Write block 464 len 36864 data: 2 2 2 2 2 2 2 2 2 04/30 Read block 436 len 12288 05/30 Read block 543 len 24576 06/30 Write block 539 len 20480 data: 2 2 2 2 3 07/30 Flush 08/30 Write block 444 len 8192 data: 4 4 09/30 Write block 404 len 16384 data: 4 3 3 3 10/30 Write block 214 len 20480 data: 4 3 3 2 2 11/30 Read block 458 len 8192 12/30 Read block 355 len 20480 13/30 Read block 588 len 20480 14/30 Flush 15/30 Flush 16/30 Write block 107 len 24576 data: 3 3 4 4 4 5 17/30 Read block 592 len 12288 18/30 Read block 294 len 40960 19/30 Read block 449 len 40960 20/30 Write block 304 len 28672 data: 5 5 7 8 7 7 8 21/30 Write block 588 len 28672 data: 4 4 4 4 4 4 4 22/30 Read block 556 len 40960 23/30 Read block 532 len 40960 24/30 Write block 557 len 12288 data: 5 3 4 25/30 Read block 149 len 16384 26/30 Read block 121 len 8192 27/30 Read block 2 len 4096 28/30 Write block 381 len 32768 data: 7 7 7 6 4 4 3 2 29/30 Read block 452 len 12288 30/30 Write block 586 len 20480 data: 3 4 5 5 5 ---------------------------------------------------------------- Crucible gen:28 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 4 Done Done Sent false 1002 Acked Write 9 Done Done Sent false 1003 Acked Read 3 Done Done Sent false 1004 Acked Read 6 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 2 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Write 5 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Read 5 Done Done Sent false 1012 Acked Read 5 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 6 Done Done Sent false 1016 Acked Read 3 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Write 7 Done Done Sent false 1021 Acked Read 10 Done Done Sent false 1022 Acked Read 10 Done Done Sent false 1023 Acked Write 3 Done Done Sent false 1024 Acked Read 4 Done Done Sent false 1025 Acked Read 2 Done Done Sent false 1026 Acked Read 1 Done Done Sent false 1027 Acked Write 8 Done Done Sent false 1028 Acked Read 3 Done Done Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 29 31 Done 29 29 1 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1014)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:17.489279309Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:17.489458386Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1245,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 46 46 46 F F F 1 020-039 27 27 27 49 49 49 F F F 2 040-059 27 27 27 49 49 49 F F F 3 060-079 24 24 24 46 46 46 F F F 4 080-099 26 26 26 48 48 48 F F F 5 100-119 24 24 24 46 46 46 T T F <--- 6 120-139 21 21 21 39 39 39 F F F 7 140-159 28 28 26 50 50 48 F F T <--- 8 160-179 22 22 22 43 43 43 F F F 9 180-199 21 21 21 39 39 39 F F F 10 200-219 28 28 22 51 51 40 F F F <--- 11 220-239 26 26 26 48 48 48 F F F 12 240-259 27 27 27 49 49 49 F F F 13 260-279 27 27 27 49 49 49 F F F 14 280-299 23 23 23 44 44 44 F F F 15 300-319 26 26 26 48 48 48 T T F <--- 16 320-339 25 25 25 47 47 47 F F F 17 340-359 27 27 27 49 49 49 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 27 27 27 49 49 49 T T F <--- 20 400-419 28 28 18 51 51 32 F F F <--- 21 420-439 4 4 4 9 9 9 F F F 22 440-459 28 28 15 51 51 29 F F F <--- 23 460-479 28 28 22 50 50 42 F F F <--- 24 480-499 24 24 24 45 45 45 F F F 25 500-519 26 26 26 48 48 48 F F F 26 520-539 28 28 26 50 50 48 F F F <--- 27 540-559 28 28 24 50 50 46 T T F <--- 28 560-579 24 24 24 46 46 46 F F F 29 580-599 22 22 22 42 42 42 T T F <--- Max gen: 28, Max flush: 51 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 14 Verifying data now No change after verify Loop: 14 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 24 24 46 46 46 F F F 1 020-039 27 27 27 49 49 49 F F F 2 040-059 27 27 27 49 49 49 F F F 3 060-079 24 24 24 46 46 46 F F F 4 080-099 26 26 26 48 48 48 F F F 5 100-119 29 29 29 52 52 52 F F F 6 120-139 21 21 21 39 39 39 F F F 7 140-159 28 28 28 50 50 50 F F F 8 160-179 22 22 22 43 43 43 F F F 9 180-199 21 21 21 39 39 39 F F F 10 200-219 28 28 28 51 51 51 F F F 11 220-239 26 26 26 48 48 48 F F F 12 240-259 27 27 27 49 49 49 F F F 13 260-279 27 27 27 49 49 49 F F F 14 280-299 23 23 23 44 44 44 F F F 15 300-319 29 29 29 52 52 52 F F F 16 320-339 25 25 25 47 47 47 F F F 17 340-359 27 27 27 49 49 49 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 29 29 29 52 52 52 F F F 20 400-419 28 28 28 51 51 51 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 28 28 28 51 51 51 F F F 23 460-479 28 28 28 50 50 50 F F F 24 480-499 24 24 24 45 45 45 F F F 25 500-519 26 26 26 48 48 48 F F F 26 520-539 28 28 28 50 50 50 F F F 27 540-559 29 29 29 52 52 52 F F F 28 560-579 24 24 24 46 46 46 F F F 29 580-599 29 29 29 52 52 52 F F F Max gen: 29, Max flush: 52 Begin loop 15 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.734278377Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:21.741413991Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.741456223Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.748262608Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.748532928Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.7485731Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.748582089Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.748593536Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253} {"msg":"Crucible 266bb1c2-c160-4cbf-8b56-b38089e1c5d5 has session id: 7a252476-251f-4ca5-a1d1-f7ae8cf5f02c","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.748907068Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"Upstairs opts: Upstairs UUID: 266bb1c2-c160-4cbf-8b56-b38089e1c5d5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.748930903Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"Crucible stats registered with UUID: 266bb1c2-c160-4cbf-8b56-b38089e1c5d5","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.748942741Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:21.748959808Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749111559Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"io task","client":"2","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749143413Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"io task","client":"0","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749154451Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"io task","client":"1","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749282537Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"task":"crutest"} {"msg":"The guest has requested activation with gen:30","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749296815Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253} {"msg":"266bb1c2-c160-4cbf-8b56-b38089e1c5d5 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749313991Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.74932295Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"client":"0","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749332598Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"client":"1","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749342246Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"client":"2","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749351664Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"io task","client":"2","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749361563Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"io task","client":"0","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749371371Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"io task","client":"1","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"ds_connection connected from Ok(127.0.0.1:59001)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749418282Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"io task","client":"2","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"ds_connection connected from Ok(127.0.0.1:57558)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749444007Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"io task","client":"1","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"ds_connection connected from Ok(127.0.0.1:64674)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.749550717Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"io task","client":"0","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.75006686Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"client":"1","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750079198Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"client":"0","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750089976Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"client":"2","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750373583Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750383081Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"[0]R flush_numbers[0..12]: [46, 49, 49, 46, 48, 52, 39, 50, 43, 39, 51, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750455708Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"downstairs","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"[0]R generation[0..12]: [24, 27, 27, 24, 26, 29, 21, 28, 22, 21, 28, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750466686Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"downstairs","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750477494Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"downstairs","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"[1]R flush_numbers[0..12]: [46, 49, 49, 46, 48, 52, 39, 50, 43, 39, 51, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750487892Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"downstairs","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"[1]R generation[0..12]: [24, 27, 27, 24, 26, 29, 21, 28, 22, 21, 28, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.75049799Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"downstairs","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750508708Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"downstairs","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"[2]R flush_numbers[0..12]: [46, 49, 49, 46, 48, 52, 39, 50, 43, 39, 51, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750518966Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"downstairs","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"[2]R generation[0..12]: [24, 27, 27, 24, 26, 29, 21, 28, 22, 21, 28, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750529094Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"downstairs","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750539862Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"downstairs","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"Max found gen is 30","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.75055066Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"downstairs","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"Generation requested: 30 >= found:30","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750560168Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"downstairs","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"Next flush: 53","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750569296Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"downstairs","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750578545Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"":"downstairs","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750588163Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750596841Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"client":"0","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750606409Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"client":"1","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750615888Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"client":"2","":"downstairs","session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750625626Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750634784Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"266bb1c2-c160-4cbf-8b56-b38089e1c5d5 is now active with session: 7a252476-251f-4ca5-a1d1-f7ae8cf5f02c","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750644732Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750653331Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"session_id":"7a252476-251f-4ca5-a1d1-f7ae8cf5f02c"} {"msg":"The guest has finished waiting for activation with:30","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750662329Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253} {"msg":"Activated sub_volume 266bb1c2-c160-4cbf-8b56-b38089e1c5d5","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750670277Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750678196Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.750960063Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"task":"crutest"} 01/30 Write block 18 len 4096 data: 2 02/30 Write block 163 len 24576 data: 3 2 2 2 2 2 03/30 Read block 414 len 20480 04/30 Read block 347 len 20480 05/30 Read block 441 len 36864 06/30 Flush 07/30 Write block 489 len 16384 data: 4 4 4 3 08/30 Read block 411 len 12288 09/30 Read block 163 len 36864 10/30 Write block 30 len 12288 data: 3 3 4 11/30 Write block 183 len 16384 data: 4 4 4 5 12/30 Write block 292 len 12288 data: 3 3 3 13/30 Read block 170 len 16384 14/30 Read block 515 len 12288 15/30 Write block 165 len 24576 data: 3 3 3 3 2 2 16/30 Read block 154 len 16384 17/30 Read block 406 len 8192 18/30 Flush 19/30 Write block 563 len 36864 data: 4 4 6 6 4 3 3 3 3 20/30 Flush 21/30 Write block 19 len 8192 data: 2 2 22/30 Write block 527 len 36864 data: 3 3 3 3 3 3 3 3 3 23/30 Read block 377 len 36864 24/30 Write block 269 len 12288 data: 5 4 3 25/30 Read block 433 len 4096 26/30 Write block 595 len 16384 data: 4 3 2 3 27/30 Read block 285 len 16384 28/30 Write block 587 len 36864 data: 5 6 6 6 5 5 5 5 5 29/30 Write block 327 len 36864 data: 4 4 4 4 3 4 3 3 4 30/30 Write block 440 len 16384 data: 2 3 3 4 ---------------------------------------------------------------- Crucible gen:30 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Sent Done Done false 1001 Acked Write 6 Sent Done Done false 1002 Acked Read 5 Sent Done Done false 1003 Acked Read 5 Sent Done Done false 1004 Acked Read 9 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 4 Sent Done Done false 1007 Acked Read 3 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Write 4 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Read 4 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Write 6 Sent Done Done false 1015 Acked Read 4 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Write 9 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Write 4 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Write 9 Sent Sent Sent false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 3 3 36 Done 0 27 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:21.759078603Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:21.759264188Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1253,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 24 30 30 46 53 53 F T T <--- 1 020-039 27 30 30 49 54 54 F T T <--- 2 040-059 27 27 27 49 49 49 F F F 3 060-079 24 24 24 46 46 46 F F F 4 080-099 26 26 26 48 48 48 F F F 5 100-119 29 29 29 52 52 52 F F F 6 120-139 21 21 21 39 39 39 F F F 7 140-159 28 28 28 50 50 50 F F F 8 160-179 22 30 30 43 54 54 F F F <--- 9 180-199 21 30 30 39 54 54 F F F <--- 10 200-219 28 28 28 51 51 51 F F F 11 220-239 26 26 26 48 48 48 F F F 12 240-259 27 27 27 49 49 49 F F F 13 260-279 27 27 27 49 49 49 F T T <--- 14 280-299 23 30 30 44 54 54 F F F <--- 15 300-319 29 29 29 52 52 52 F F F 16 320-339 25 25 25 47 47 47 F T T <--- 17 340-359 27 27 27 49 49 49 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 29 29 29 52 52 52 F F F 20 400-419 28 28 28 51 51 51 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 28 28 28 51 51 51 F T T <--- 23 460-479 28 28 28 50 50 50 F F F 24 480-499 24 30 30 45 54 54 F F F <--- 25 500-519 26 26 26 48 48 48 F F F 26 520-539 28 28 28 50 50 50 F T T <--- 27 540-559 29 29 29 52 52 52 F F F 28 560-579 24 30 30 46 55 55 F F F <--- 29 580-599 29 29 29 52 52 52 F T T <--- Max gen: 30, Max flush: 55 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 15 Verifying data now No change after verify Loop: 15 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 31 31 31 56 56 56 F F F 1 020-039 31 31 31 56 56 56 F F F 2 040-059 27 27 27 49 49 49 F F F 3 060-079 24 24 24 46 46 46 F F F 4 080-099 26 26 26 48 48 48 F F F 5 100-119 29 29 29 52 52 52 F F F 6 120-139 21 21 21 39 39 39 F F F 7 140-159 28 28 28 50 50 50 F F F 8 160-179 30 30 30 54 54 54 F F F 9 180-199 30 30 30 54 54 54 F F F 10 200-219 28 28 28 51 51 51 F F F 11 220-239 26 26 26 48 48 48 F F F 12 240-259 27 27 27 49 49 49 F F F 13 260-279 31 31 31 56 56 56 F F F 14 280-299 30 30 30 54 54 54 F F F 15 300-319 29 29 29 52 52 52 F F F 16 320-339 31 31 31 56 56 56 F F F 17 340-359 27 27 27 49 49 49 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 29 29 29 52 52 52 F F F 20 400-419 28 28 28 51 51 51 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 31 31 31 56 56 56 F F F 23 460-479 28 28 28 50 50 50 F F F 24 480-499 30 30 30 54 54 54 F F F 25 500-519 26 26 26 48 48 48 F F F 26 520-539 31 31 31 56 56 56 F F F 27 540-559 29 29 29 52 52 52 F F F 28 560-579 30 30 30 55 55 55 F F F 29 580-599 31 31 31 56 56 56 F F F Max gen: 31, Max flush: 56 Begin loop 16 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.009707676Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.017922837Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.018224541Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.018251816Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.018260114Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.018271512Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261} {"msg":"Crucible 826f21cb-7bfe-4d0d-a29f-e331c9056c27 has session id: 667f8d5e-006f-4b9c-889a-ce5da29774c8","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.018627765Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"Upstairs opts: Upstairs UUID: 826f21cb-7bfe-4d0d-a29f-e331c9056c27, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.01865584Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"Crucible stats registered with UUID: 826f21cb-7bfe-4d0d-a29f-e331c9056c27","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.018666838Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:26.018676086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.01881582Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"io task","client":"1","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.018846105Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"io task","client":"0","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.018857652Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"io task","client":"2","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.018983419Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"task":"crutest"} {"msg":"The guest has requested activation with gen:32","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.019006455Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261} {"msg":"826f21cb-7bfe-4d0d-a29f-e331c9056c27 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.01903208Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.019041548Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"client":"0","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.019051346Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"client":"1","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.019060984Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"client":"2","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.019070523Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"io task","client":"2","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.019080451Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"io task","client":"0","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.019090509Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"io task","client":"1","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"ds_connection connected from Ok(127.0.0.1:50773)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.019133921Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"io task","client":"1","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"ds_connection connected from Ok(127.0.0.1:58449)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.019234202Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"io task","client":"2","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"ds_connection connected from Ok(127.0.0.1:50968)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.019260117Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"io task","client":"0","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.019839279Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"client":"0","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.019855985Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"client":"1","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.019866793Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"client":"2","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020171746Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020192582Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"[0]R flush_numbers[0..12]: [56, 56, 49, 46, 48, 52, 39, 50, 54, 54, 51, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020281326Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"downstairs","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"[0]R generation[0..12]: [31, 31, 27, 24, 26, 29, 21, 28, 30, 30, 28, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020292744Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"downstairs","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020303742Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"downstairs","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"[1]R flush_numbers[0..12]: [56, 56, 49, 46, 48, 52, 39, 50, 54, 54, 51, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.02031419Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"downstairs","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"[1]R generation[0..12]: [31, 31, 27, 24, 26, 29, 21, 28, 30, 30, 28, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020324878Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"downstairs","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020335676Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"downstairs","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"[2]R flush_numbers[0..12]: [56, 56, 49, 46, 48, 52, 39, 50, 54, 54, 51, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020346044Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"downstairs","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"[2]R generation[0..12]: [31, 31, 27, 24, 26, 29, 21, 28, 30, 30, 28, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020356422Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"downstairs","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.02036737Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"downstairs","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"Max found gen is 32","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020376908Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"downstairs","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"Generation requested: 32 >= found:32","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020386696Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"downstairs","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"Next flush: 57","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020395994Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"downstairs","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020405293Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"":"downstairs","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020414961Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020423639Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"client":"0","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020434737Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"client":"1","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020444215Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"client":"2","":"downstairs","session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020453943Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020463002Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"826f21cb-7bfe-4d0d-a29f-e331c9056c27 is now active with session: 667f8d5e-006f-4b9c-889a-ce5da29774c8","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020479429Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020488177Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"session_id":"667f8d5e-006f-4b9c-889a-ce5da29774c8"} {"msg":"The guest has finished waiting for activation with:32","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020497375Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261} {"msg":"Activated sub_volume 826f21cb-7bfe-4d0d-a29f-e331c9056c27","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020505654Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020513882Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"task":"crutest"} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.020728832Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"task":"crutest"} 01/30 Write block 240 len 24576 data: 2 3 3 3 3 3 02/30 Read block 133 len 12288 03/30 Read block 66 len 24576 04/30 Read block 68 len 36864 05/30 Read block 65 len 8192 06/30 Flush 07/30 Write block 209 len 40960 data: 5 4 3 4 4 5 4 4 3 3 08/30 Write block 137 len 16384 data: 3 3 3 3 09/30 Read block 305 len 32768 10/30 Write block 580 len 16384 data: 2 2 3 2 11/30 Read block 230 len 4096 12/30 Write block 592 len 24576 data: 6 6 6 6 4 3 13/30 Write block 18 len 36864 data: 3 3 3 2 2 2 2 2 4 14/30 Write block 559 len 20480 data: 5 4 4 4 5 15/30 Flush 16/30 Write block 343 len 36864 data: 3 3 2 2 2 2 2 2 2 17/30 Read block 481 len 4096 18/30 Flush 19/30 Write block 302 len 36864 data: 3 4 6 6 8 9 8 8 9 20/30 Flush 21/30 Write block 267 len 16384 data: 5 5 6 5 22/30 Read block 349 len 16384 23/30 Write block 275 len 24576 data: 3 3 3 2 2 2 24/30 Write block 208 len 8192 data: 5 6 25/30 Write block 143 len 28672 data: 3 4 4 3 4 2 2 26/30 Read block 564 len 8192 27/30 Read block 483 len 4096 28/30 Write block 338 len 32768 data: 3 3 3 3 3 4 4 3 29/30 Write block 121 len 16384 data: 5 4 7 9 30/30 Read block 488 len 16384 ---------------------------------------------------------------- Crucible gen:32 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 3 Done Sent Done false 1002 Acked Read 6 Done Sent Done false 1003 Acked Read 9 Done Sent Done false 1004 Acked Read 2 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 10 Done Sent Done false 1007 Acked Write 4 Done Sent Done false 1008 Acked Read 8 Done Sent Done false 1009 Acked Write 4 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Write 9 Done Sent Done false 1013 Acked Write 5 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 9 Done Sent Done false 1016 Acked Read 1 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 9 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 4 Done Sent Done false 1021 Acked Read 4 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Write 2 Done Sent Done false 1024 Acked Write 7 Done Sent Done false 1025 Acked Read 2 Done Sent Done false 1026 Acked Read 1 Done Sent Done false 1027 Acked Write 8 Done Sent Done false 1028 Acked Write 4 Done Sent Done false 1029 Acked Read 4 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 0 29 Done 30 1 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:26.029501609Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:26.029676966Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1261,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 31 32 58 56 58 F F F <--- 1 020-039 32 31 32 58 56 58 F F F <--- 2 040-059 27 27 27 49 49 49 F F F 3 060-079 24 24 24 46 46 46 F F F 4 080-099 26 26 26 48 48 48 F F F 5 100-119 29 29 29 52 52 52 F F F 6 120-139 32 21 32 58 39 58 T F T <--- 7 140-159 32 28 32 58 50 58 T F T <--- 8 160-179 30 30 30 54 54 54 F F F 9 180-199 30 30 30 54 54 54 F F F 10 200-219 32 28 32 58 51 58 T F T <--- 11 220-239 26 26 26 48 48 48 F F F 12 240-259 32 27 32 57 49 57 F T F <--- 13 260-279 31 31 31 56 56 56 T F T <--- 14 280-299 30 30 30 54 54 54 T F T <--- 15 300-319 32 29 32 60 52 60 F F F <--- 16 320-339 31 31 31 56 56 56 T F T <--- 17 340-359 32 27 32 59 49 59 T F T <--- 18 360-379 27 27 27 49 49 49 F F F 19 380-399 29 29 29 52 52 52 F F F 20 400-419 28 28 28 51 51 51 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 31 31 31 56 56 56 F F F 23 460-479 28 28 28 50 50 50 F F F 24 480-499 30 30 30 54 54 54 F F F 25 500-519 26 26 26 48 48 48 F F F 26 520-539 31 31 31 56 56 56 F F F 27 540-559 32 29 32 58 52 58 F F F <--- 28 560-579 32 30 32 58 55 58 F F F <--- 29 580-599 32 31 32 58 56 58 F F F <--- Max gen: 32, Max flush: 60 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 16 Verifying data now No change after verify Loop: 16 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 58 58 58 F F F 1 020-039 32 32 32 58 58 58 F F F 2 040-059 27 27 27 49 49 49 F F F 3 060-079 24 24 24 46 46 46 F F F 4 080-099 26 26 26 48 48 48 F F F 5 100-119 29 29 29 52 52 52 F F F 6 120-139 33 33 33 61 61 61 F F F 7 140-159 33 33 33 61 61 61 F F F 8 160-179 30 30 30 54 54 54 F F F 9 180-199 30 30 30 54 54 54 F F F 10 200-219 33 33 33 61 61 61 F F F 11 220-239 26 26 26 48 48 48 F F F 12 240-259 32 32 32 57 57 57 F F F 13 260-279 33 33 33 61 61 61 F F F 14 280-299 33 33 33 61 61 61 F F F 15 300-319 32 32 32 60 60 60 F F F 16 320-339 33 33 33 61 61 61 F F F 17 340-359 33 33 33 61 61 61 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 29 29 29 52 52 52 F F F 20 400-419 28 28 28 51 51 51 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 31 31 31 56 56 56 F F F 23 460-479 28 28 28 50 50 50 F F F 24 480-499 30 30 30 54 54 54 F F F 25 500-519 26 26 26 48 48 48 F F F 26 520-539 31 31 31 56 56 56 F F F 27 540-559 32 32 32 58 58 58 F F F 28 560-579 32 32 32 58 58 58 F F F 29 580-599 32 32 32 58 58 58 F F F Max gen: 33, Max flush: 61 Begin loop 17 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.277773144Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.285863329Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.286161144Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.286190068Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.286198697Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.286212314Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269} {"msg":"Crucible bf8277ea-9854-420e-b1f6-f11426a9cf53 has session id: eb5adca0-829b-435e-b4ab-88213fc2b313","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.286567158Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"Upstairs opts: Upstairs UUID: bf8277ea-9854-420e-b1f6-f11426a9cf53, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.286597502Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"Crucible stats registered with UUID: bf8277ea-9854-420e-b1f6-f11426a9cf53","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.28660869Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:30.286617988Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.286764051Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"io task","client":"1","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.286794755Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"io task","client":"0","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.286805833Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"io task","client":"2","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.286942437Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"task":"crutest"} {"msg":"The guest has requested activation with gen:34","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.286957904Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269} {"msg":"bf8277ea-9854-420e-b1f6-f11426a9cf53 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.286990798Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.287000166Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"client":"0","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.287009955Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"client":"1","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.287019413Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"client":"2","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.287028751Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"io task","client":"2","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.287040809Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"io task","client":"0","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.287067244Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"io task","client":"1","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"ds_connection connected from Ok(127.0.0.1:36972)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.287122174Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"io task","client":"0","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"ds_connection connected from Ok(127.0.0.1:47841)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.287134751Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"io task","client":"2","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"ds_connection connected from Ok(127.0.0.1:37969)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.287145079Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"io task","client":"1","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.287858276Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"client":"0","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.287882841Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"client":"2","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.287920674Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"client":"1","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288201981Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288250612Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"[0]R flush_numbers[0..12]: [58, 58, 49, 46, 48, 52, 61, 61, 54, 54, 61, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288289615Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"downstairs","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"[0]R generation[0..12]: [32, 32, 27, 24, 26, 29, 33, 33, 30, 30, 33, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288300653Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"downstairs","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288311561Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"downstairs","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"[1]R flush_numbers[0..12]: [58, 58, 49, 46, 48, 52, 61, 61, 54, 54, 61, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288321949Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"downstairs","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"[1]R generation[0..12]: [32, 32, 27, 24, 26, 29, 33, 33, 30, 30, 33, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288332317Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"downstairs","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288345794Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"downstairs","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"[2]R flush_numbers[0..12]: [58, 58, 49, 46, 48, 52, 61, 61, 54, 54, 61, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288356103Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"downstairs","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"[2]R generation[0..12]: [32, 32, 27, 24, 26, 29, 33, 33, 30, 30, 33, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288366311Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"downstairs","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288377169Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"downstairs","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"Max found gen is 34","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288386597Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"downstairs","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"Generation requested: 34 >= found:34","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288396215Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"downstairs","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"Next flush: 62","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288405483Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"downstairs","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288414812Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"":"downstairs","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.28842476Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288433748Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"client":"0","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288443396Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"client":"1","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288452854Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"client":"2","":"downstairs","session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288462423Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288471361Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"bf8277ea-9854-420e-b1f6-f11426a9cf53 is now active with session: eb5adca0-829b-435e-b4ab-88213fc2b313","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288481119Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"The guest has finished waiting for activation with:34","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288491237Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288498686Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"session_id":"eb5adca0-829b-435e-b4ab-88213fc2b313"} {"msg":"Activated sub_volume bf8277ea-9854-420e-b1f6-f11426a9cf53","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288513003Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288521052Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.288840602Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"task":"crutest"} 01/30 Write block 23 len 16384 data: 3 3 3 5 02/30 Read block 374 len 8192 03/30 Write block 72 len 28672 data: 2 2 2 3 3 3 4 04/30 Read block 67 len 16384 05/30 Read block 304 len 32768 06/30 Write block 554 len 40960 data: 4 4 4 6 4 6 5 5 5 6 07/30 Write block 20 len 28672 data: 4 3 3 4 4 4 6 08/30 Read block 308 len 36864 09/30 Flush 10/30 Write block 489 len 40960 data: 5 5 5 4 3 3 3 3 2 3 11/30 Read block 329 len 28672 12/30 Read block 409 len 28672 13/30 Read block 390 len 16384 14/30 Write block 132 len 12288 data: 5 5 3 15/30 Read block 13 len 8192 16/30 Write block 279 len 4096 data: 3 17/30 Read block 378 len 28672 18/30 Write block 245 len 36864 data: 4 3 3 3 3 2 3 4 5 19/30 Write block 138 len 20480 data: 4 4 4 4 4 20/30 Flush 21/30 Write block 324 len 12288 data: 4 3 4 22/30 Write block 349 len 36864 data: 3 3 3 2 2 2 2 3 3 23/30 Write block 289 len 24576 data: 4 4 4 4 4 4 24/30 Read block 456 len 4096 25/30 Read block 38 len 8192 26/30 Read block 297 len 36864 27/30 Read block 276 len 8192 28/30 Read block 554 len 36864 29/30 Write block 16 len 40960 data: 4 3 4 4 5 4 4 5 5 5 30/30 Read block 199 len 8192 ---------------------------------------------------------------- Crucible gen:34 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Sent Done false 1001 Acked Read 2 Done Sent Done false 1002 Acked Write 7 Done Sent Done false 1003 Acked Read 4 Done Sent Done false 1004 Acked Read 8 Done Sent Done false 1005 Acked Write 10 Done Sent Done false 1006 Acked Write 7 Done Sent Done false 1007 Acked Read 9 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 10 Done Sent Done false 1010 Acked Read 7 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Read 2 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Read 7 Done Sent Done false 1017 Acked Write 9 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 3 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Read 1 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Read 9 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Write 10 Sent Sent Done false 1029 Acked Read 2 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 30 0 32 Done 28 0 30 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:30.298478947Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:30.298634927Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1269,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 32 32 32 58 58 58 T F T <--- 1 020-039 34 32 34 62 58 62 T F T <--- 2 040-059 27 27 27 49 49 49 F F F 3 060-079 34 24 34 62 46 62 F F F <--- 4 080-099 26 26 26 48 48 48 F F F 5 100-119 29 29 29 52 52 52 F F F 6 120-139 34 33 34 63 61 63 F F F <--- 7 140-159 34 33 34 63 61 63 F F F <--- 8 160-179 30 30 30 54 54 54 F F F 9 180-199 30 30 30 54 54 54 F F F 10 200-219 33 33 33 61 61 61 F F F 11 220-239 26 26 26 48 48 48 F F F 12 240-259 34 32 34 63 57 63 F F F <--- 13 260-279 34 33 34 63 61 63 F F F <--- 14 280-299 33 33 33 61 61 61 T F T <--- 15 300-319 32 32 32 60 60 60 F F F 16 320-339 33 33 33 61 61 61 T F T <--- 17 340-359 33 33 33 61 61 61 T F T <--- 18 360-379 27 27 27 49 49 49 F F F 19 380-399 29 29 29 52 52 52 F F F 20 400-419 28 28 28 51 51 51 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 31 31 31 56 56 56 F F F 23 460-479 28 28 28 50 50 50 F F F 24 480-499 34 30 34 63 54 63 F F F <--- 25 500-519 26 26 26 48 48 48 F F F 26 520-539 31 31 31 56 56 56 F F F 27 540-559 34 32 34 62 58 62 F F F <--- 28 560-579 34 32 34 62 58 62 F F F <--- 29 580-599 32 32 32 58 58 58 F F F Max gen: 34, Max flush: 63 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 17 Verifying data now No change after verify Loop: 17 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 64 64 64 F F F 1 020-039 35 35 35 64 64 64 F F F 2 040-059 27 27 27 49 49 49 F F F 3 060-079 34 34 34 62 62 62 F F F 4 080-099 26 26 26 48 48 48 F F F 5 100-119 29 29 29 52 52 52 F F F 6 120-139 34 34 34 63 63 63 F F F 7 140-159 34 34 34 63 63 63 F F F 8 160-179 30 30 30 54 54 54 F F F 9 180-199 30 30 30 54 54 54 F F F 10 200-219 33 33 33 61 61 61 F F F 11 220-239 26 26 26 48 48 48 F F F 12 240-259 34 34 34 63 63 63 F F F 13 260-279 34 34 34 63 63 63 F F F 14 280-299 35 35 35 64 64 64 F F F 15 300-319 32 32 32 60 60 60 F F F 16 320-339 35 35 35 64 64 64 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 29 29 29 52 52 52 F F F 20 400-419 28 28 28 51 51 51 F F F 21 420-439 4 4 4 9 9 9 F F F 22 440-459 31 31 31 56 56 56 F F F 23 460-479 28 28 28 50 50 50 F F F 24 480-499 34 34 34 63 63 63 F F F 25 500-519 26 26 26 48 48 48 F F F 26 520-539 31 31 31 56 56 56 F F F 27 540-559 34 34 34 62 62 62 F F F 28 560-579 34 34 34 62 62 62 F F F 29 580-599 32 32 32 58 58 58 F F F Max gen: 35, Max flush: 64 Begin loop 18 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.545681572Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:34.55273954Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.552781093Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.559665473Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.559937172Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.559964627Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.559973236Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.559984684Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277} {"msg":"Crucible 5bb5f5ab-8171-4d53-af83-ae93cd3d77c6 has session id: 9908c170-bb48-4a28-abea-5ac811c76ba2","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560290196Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"Upstairs opts: Upstairs UUID: 5bb5f5ab-8171-4d53-af83-ae93cd3d77c6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560309683Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"Crucible stats registered with UUID: 5bb5f5ab-8171-4d53-af83-ae93cd3d77c6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.56032138Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:34.560330799Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560478641Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"io task","client":"1","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560494888Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"io task","client":"0","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560505016Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"io task","client":"2","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560628663Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"task":"crutest"} {"msg":"The guest has requested activation with gen:36","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560641041Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277} {"msg":"5bb5f5ab-8171-4d53-af83-ae93cd3d77c6 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560663946Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560673195Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"client":"0","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560682783Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"client":"1","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560692321Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"client":"2","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560701779Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"io task","client":"2","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560711477Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"io task","client":"0","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560721336Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"io task","client":"1","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"ds_connection connected from Ok(127.0.0.1:62092)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560766447Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"io task","client":"2","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"ds_connection connected from Ok(127.0.0.1:56104)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560777175Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"io task","client":"1","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"ds_connection connected from Ok(127.0.0.1:49226)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.560796102Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"io task","client":"0","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.561493291Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"client":"1","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.561526265Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"client":"0","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.56155304Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"client":"2","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.561802573Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.561844395Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"[0]R flush_numbers[0..12]: [64, 64, 49, 62, 48, 52, 63, 63, 54, 54, 61, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.561894386Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"downstairs","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"[0]R generation[0..12]: [35, 35, 27, 34, 26, 29, 34, 34, 30, 30, 33, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.561905414Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"downstairs","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.561916282Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"downstairs","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"[1]R flush_numbers[0..12]: [64, 64, 49, 62, 48, 52, 63, 63, 54, 54, 61, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.56192652Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"downstairs","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"[1]R generation[0..12]: [35, 35, 27, 34, 26, 29, 34, 34, 30, 30, 33, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.561936718Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"downstairs","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.561947576Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"downstairs","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"[2]R flush_numbers[0..12]: [64, 64, 49, 62, 48, 52, 63, 63, 54, 54, 61, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.561957884Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"downstairs","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"[2]R generation[0..12]: [35, 35, 27, 34, 26, 29, 34, 34, 30, 30, 33, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.561968222Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"downstairs","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.56198166Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"downstairs","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"Max found gen is 36","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.561991068Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"downstairs","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"Generation requested: 36 >= found:36","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.562000666Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"downstairs","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"Next flush: 65","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.562010014Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"downstairs","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.562019302Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"":"downstairs","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.562029361Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.562038079Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"client":"0","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.562047917Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"client":"1","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.562057815Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"client":"2","":"downstairs","session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.562067833Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.562076972Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"5bb5f5ab-8171-4d53-af83-ae93cd3d77c6 is now active with session: 9908c170-bb48-4a28-abea-5ac811c76ba2","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.56208678Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.562095478Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"session_id":"9908c170-bb48-4a28-abea-5ac811c76ba2"} {"msg":"The guest has finished waiting for activation with:36","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.562104377Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277} {"msg":"Activated sub_volume 5bb5f5ab-8171-4d53-af83-ae93cd3d77c6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.562112265Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.562120274Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.56240426Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"task":"crutest"} 01/30 Write block 274 len 8192 data: 3 4 02/30 Write block 593 len 24576 data: 7 7 7 5 4 4 03/30 Read block 373 len 36864 04/30 Flush 05/30 Write block 42 len 32768 data: 4 4 4 4 4 3 3 3 06/30 Read block 565 len 16384 07/30 Write block 501 len 36864 data: 4 4 5 5 6 5 5 2 2 08/30 Flush 09/30 Write block 479 len 36864 data: 3 3 3 5 5 4 4 5 5 10/30 Write block 331 len 24576 data: 4 5 4 4 5 4 11/30 Write block 302 len 12288 data: 4 5 7 12/30 Read block 583 len 24576 13/30 Write block 495 len 24576 data: 4 4 3 4 4 4 14/30 Flush 15/30 Write block 182 len 12288 data: 4 5 5 16/30 Read block 304 len 20480 17/30 Read block 296 len 20480 18/30 Read block 28 len 32768 19/30 Write block 114 len 8192 data: 5 6 20/30 Flush 21/30 Write block 520 len 40960 data: 3 3 3 3 4 4 3 4 4 4 22/30 Write block 510 len 40960 data: 2 2 2 2 2 4 5 5 4 3 23/30 Write block 338 len 4096 data: 4 24/30 Write block 440 len 4096 data: 3 25/30 Write block 558 len 20480 data: 5 7 6 6 6 26/30 Read block 294 len 36864 27/30 Write block 427 len 8192 data: 3 3 28/30 Read block 39 len 16384 29/30 Read block 229 len 32768 30/30 Read block 305 len 16384 ---------------------------------------------------------------- Crucible gen:36 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Read 9 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 8 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Write 9 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 9 Sent Done Done false 1009 Acked Write 6 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Read 6 Sent Done Done false 1012 Acked Write 6 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 3 Sent Done Done false 1015 Acked Read 5 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Read 8 Sent Done Done false 1018 Acked Write 2 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 10 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Write 5 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Write 2 Sent Done Done false 1027 Acked Read 4 Sent Done Done false 1028 Acked Read 8 Sent Done Done false 1029 Acked Read 4 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 0 28 Done 2 30 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:34.571052331Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:34.571243765Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1277,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 64 64 64 F F F 1 020-039 35 35 35 64 64 64 F F F 2 040-059 27 36 36 49 66 66 F F F <--- 3 060-079 34 34 34 62 62 62 F F F 4 080-099 26 26 26 48 48 48 F F F 5 100-119 29 36 36 52 68 68 F F F <--- 6 120-139 34 34 34 63 63 63 F F F 7 140-159 34 34 34 63 63 63 F F F 8 160-179 30 30 30 54 54 54 F F F 9 180-199 30 36 36 54 68 68 F F F <--- 10 200-219 33 33 33 61 61 61 F F F 11 220-239 26 26 26 48 48 48 F F F 12 240-259 34 34 34 63 63 63 F F F 13 260-279 34 36 36 63 65 65 T F F <--- 14 280-299 35 35 35 64 64 64 F F F 15 300-319 32 36 36 60 67 67 F F F <--- 16 320-339 35 36 36 64 67 67 F T T <--- 17 340-359 35 35 35 64 64 64 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 29 29 29 52 52 52 F F F 20 400-419 28 28 28 51 51 51 F F F 21 420-439 4 4 4 9 9 9 F T T <--- 22 440-459 31 31 31 56 56 56 F T T <--- 23 460-479 28 36 36 50 67 67 F F F <--- 24 480-499 34 36 36 63 67 67 F F F <--- 25 500-519 26 36 36 48 67 67 F T T <--- 26 520-539 31 31 31 56 56 56 F T T <--- 27 540-559 34 34 34 62 62 62 F T T <--- 28 560-579 34 34 34 62 62 62 F T T <--- 29 580-599 32 36 36 58 65 65 T F F <--- Max gen: 36, Max flush: 68 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 18 Verifying data now No change after verify Loop: 18 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 64 64 64 F F F 1 020-039 35 35 35 64 64 64 F F F 2 040-059 36 36 36 66 66 66 F F F 3 060-079 34 34 34 62 62 62 F F F 4 080-099 26 26 26 48 48 48 F F F 5 100-119 36 36 36 68 68 68 F F F 6 120-139 34 34 34 63 63 63 F F F 7 140-159 34 34 34 63 63 63 F F F 8 160-179 30 30 30 54 54 54 F F F 9 180-199 36 36 36 68 68 68 F F F 10 200-219 33 33 33 61 61 61 F F F 11 220-239 26 26 26 48 48 48 F F F 12 240-259 34 34 34 63 63 63 F F F 13 260-279 36 36 36 65 65 65 F F F 14 280-299 35 35 35 64 64 64 F F F 15 300-319 36 36 36 67 67 67 F F F 16 320-339 37 37 37 69 69 69 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 29 29 29 52 52 52 F F F 20 400-419 28 28 28 51 51 51 F F F 21 420-439 37 37 37 69 69 69 F F F 22 440-459 37 37 37 69 69 69 F F F 23 460-479 36 36 36 67 67 67 F F F 24 480-499 36 36 36 67 67 67 F F F 25 500-519 37 37 37 69 69 69 F F F 26 520-539 37 37 37 69 69 69 F F F 27 540-559 37 37 37 69 69 69 F F F 28 560-579 37 37 37 69 69 69 F F F 29 580-599 36 36 36 65 65 65 F F F Max gen: 37, Max flush: 69 Begin loop 19 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.828822197Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:38.835936834Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.835991254Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.842750828Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843033395Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.84306044Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843069058Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843080476Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285} {"msg":"Crucible bcd54bbc-7f50-4932-9205-900228edfdda has session id: 6339d830-59e5-46c4-8f72-d9e754c24455","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.84343244Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Upstairs opts: Upstairs UUID: bcd54bbc-7f50-4932-9205-900228edfdda, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843461745Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Crucible stats registered with UUID: bcd54bbc-7f50-4932-9205-900228edfdda","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843473163Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:38.843482631Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843633613Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"io task","client":"1","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843651769Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"io task","client":"0","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843663947Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"io task","client":"2","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843782875Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"task":"crutest"} {"msg":"The guest has requested activation with gen:38","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843799502Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285} {"msg":"bcd54bbc-7f50-4932-9205-900228edfdda active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843832605Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843842044Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"client":"0","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843851762Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"client":"1","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.84386133Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"client":"2","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843870798Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"io task","client":"2","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843880586Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"io task","client":"0","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843890535Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"io task","client":"1","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"ds_connection connected from Ok(127.0.0.1:63191)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843962651Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"io task","client":"0","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"ds_connection connected from Ok(127.0.0.1:52395)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843988076Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"io task","client":"1","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"ds_connection connected from Ok(127.0.0.1:34372)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.843999304Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"io task","client":"2","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.844689215Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"client":"1","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.844711811Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"client":"2","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.844777308Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"client":"0","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.844990259Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.8450378Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"[0]R flush_numbers[0..12]: [64, 64, 66, 62, 48, 68, 63, 63, 54, 68, 61, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845108266Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"downstairs","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"[0]R generation[0..12]: [35, 35, 36, 34, 26, 36, 34, 34, 30, 36, 33, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845119414Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"downstairs","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845130302Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"downstairs","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"[1]R flush_numbers[0..12]: [64, 64, 66, 62, 48, 68, 63, 63, 54, 68, 61, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.84514063Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"downstairs","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"[1]R generation[0..12]: [35, 35, 36, 34, 26, 36, 34, 34, 30, 36, 33, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845151128Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"downstairs","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845161826Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"downstairs","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"[2]R flush_numbers[0..12]: [64, 64, 66, 62, 48, 68, 63, 63, 54, 68, 61, 48]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845172085Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"downstairs","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"[2]R generation[0..12]: [35, 35, 36, 34, 26, 36, 34, 34, 30, 36, 33, 26]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845182283Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"downstairs","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845192961Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"downstairs","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Max found gen is 38","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845202269Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"downstairs","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Generation requested: 38 >= found:38","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845211847Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"downstairs","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Next flush: 70","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845221045Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"downstairs","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845231823Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"":"downstairs","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845241581Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.84525028Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"client":"0","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845259968Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"client":"1","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845269616Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"client":"2","":"downstairs","session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845279224Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845288313Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 bcd54bbc-7f50-4932-9205-900228edfdda is now active with session: 6339d830-59e5-46c4-8f72-d9e754c24455","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.84530398Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845312728Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"session_id":"6339d830-59e5-46c4-8f72-d9e754c24455"} {"msg":"The guest has finished waiting for activation with:38","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845321706Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285} {"msg":"Activated sub_volume bcd54bbc-7f50-4932-9205-900228edfdda","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845329785Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845337863Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"task":"crutest"} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.845552213Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"task":"crutest"} 01/30 Write block 76 len 4096 data: 4 02/30 Read block 508 len 40960 03/30 Read block 444 len 8192 04/30 Read block 579 len 28672 05/30 Read block 542 len 4096 06/30 Read block 250 len 16384 07/30 Read block 255 len 12288 08/30 Read block 523 len 16384 09/30 Read block 335 len 28672 10/30 Read block 241 len 24576 11/30 Read block 445 len 28672 12/30 Read block 507 len 12288 13/30 Write block 543 len 28672 data: 4 3 4 3 3 3 3 14/30 Read block 581 len 40960 15/30 Write block 557 len 8192 data: 7 6 16/30 Flush 17/30 Write block 118 len 4096 data: 5 18/30 Read block 404 len 8192 19/30 Read block 339 len 12288 20/30 Read block 536 len 20480 21/30 Write block 221 len 4096 data: 4 22/30 Write block 48 len 16384 data: 4 4 3 3 23/30 Read block 322 len 20480 24/30 Write block 467 len 28672 data: 3 3 3 3 3 3 2 25/30 Read block 187 len 28672 26/30 Read block 463 len 4096 27/30 Write block 295 len 16384 data: 3 3 2 2 28/30 Read block 85 len 8192 29/30 Read block 436 len 40960 30/30 Read block 466 len 24576 ---------------------------------------------------------------- Crucible gen:38 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Sent Done Done false 1001 Acked Read 10 Sent Done Done false 1002 Acked Read 2 Sent Done Done false 1003 Acked Read 7 Sent Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Read 3 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Read 6 Sent Done Done false 1010 Acked Read 7 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Write 2 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Read 2 Sent Done Done false 1018 Acked Read 3 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Write 1 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Read 5 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Read 7 Sent Done Done false 1025 Acked Read 1 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Read 6 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:38.853946441Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:38.854127137Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1285,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 64 64 64 F F F 1 020-039 35 35 35 64 64 64 F F F 2 040-059 36 36 36 66 66 66 F T T <--- 3 060-079 34 38 38 62 70 70 F F F <--- 4 080-099 26 26 26 48 48 48 F F F 5 100-119 36 36 36 68 68 68 F T T <--- 6 120-139 34 34 34 63 63 63 F F F 7 140-159 34 34 34 63 63 63 F F F 8 160-179 30 30 30 54 54 54 F F F 9 180-199 36 36 36 68 68 68 F F F 10 200-219 33 33 33 61 61 61 F F F 11 220-239 26 26 26 48 48 48 F T T <--- 12 240-259 34 34 34 63 63 63 F F F 13 260-279 36 36 36 65 65 65 F F F 14 280-299 35 35 35 64 64 64 F T T <--- 15 300-319 36 36 36 67 67 67 F F F 16 320-339 37 37 37 69 69 69 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 29 29 29 52 52 52 F F F 20 400-419 28 28 28 51 51 51 F F F 21 420-439 37 37 37 69 69 69 F F F 22 440-459 37 37 37 69 69 69 F F F 23 460-479 36 36 36 67 67 67 F T T <--- 24 480-499 36 36 36 67 67 67 F F F 25 500-519 37 37 37 69 69 69 F F F 26 520-539 37 37 37 69 69 69 F F F 27 540-559 37 38 38 69 70 70 F F F <--- 28 560-579 37 37 37 69 69 69 F F F 29 580-599 36 36 36 65 65 65 F F F Max gen: 38, Max flush: 70 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 19 Verifying data now No change after verify Loop: 19 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 64 64 64 F F F 1 020-039 35 35 35 64 64 64 F F F 2 040-059 39 39 39 71 71 71 F F F 3 060-079 38 38 38 70 70 70 F F F 4 080-099 26 26 26 48 48 48 F F F 5 100-119 39 39 39 71 71 71 F F F 6 120-139 34 34 34 63 63 63 F F F 7 140-159 34 34 34 63 63 63 F F F 8 160-179 30 30 30 54 54 54 F F F 9 180-199 36 36 36 68 68 68 F F F 10 200-219 33 33 33 61 61 61 F F F 11 220-239 39 39 39 71 71 71 F F F 12 240-259 34 34 34 63 63 63 F F F 13 260-279 36 36 36 65 65 65 F F F 14 280-299 39 39 39 71 71 71 F F F 15 300-319 36 36 36 67 67 67 F F F 16 320-339 37 37 37 69 69 69 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 29 29 29 52 52 52 F F F 20 400-419 28 28 28 51 51 51 F F F 21 420-439 37 37 37 69 69 69 F F F 22 440-459 37 37 37 69 69 69 F F F 23 460-479 39 39 39 71 71 71 F F F 24 480-499 36 36 36 67 67 67 F F F 25 500-519 37 37 37 69 69 69 F F F 26 520-539 37 37 37 69 69 69 F F F 27 540-559 38 38 38 70 70 70 F F F 28 560-579 37 37 37 69 69 69 F F F 29 580-599 36 36 36 65 65 65 F F F Max gen: 39, Max flush: 71 Begin loop 20 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.094356989Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.102610143Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.102908017Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.102935652Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.10294626Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.102957948Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293} {"msg":"Crucible a886f3b6-219b-4813-bda2-a4085eb69095 has session id: e67711fe-a9c1-47f5-9bc6-5689a0bcda16","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.10331953Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"Upstairs opts: Upstairs UUID: a886f3b6-219b-4813-bda2-a4085eb69095, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103342756Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"Crucible stats registered with UUID: a886f3b6-219b-4813-bda2-a4085eb69095","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103352284Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:43.103361842Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103511574Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"io task","client":"1","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103543578Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"io task","client":"0","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103554636Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"io task","client":"2","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103676773Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"task":"crutest"} {"msg":"The guest has requested activation with gen:40","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.10369572Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293} {"msg":"a886f3b6-219b-4813-bda2-a4085eb69095 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103722605Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103731973Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"client":"0","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103741541Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"client":"1","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103751039Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"client":"2","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103762207Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"io task","client":"2","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103772165Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"io task","client":"0","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103781944Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"io task","client":"1","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"ds_connection connected from Ok(127.0.0.1:53860)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103841902Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"io task","client":"1","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"ds_connection connected from Ok(127.0.0.1:52870)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103867238Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"io task","client":"2","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"ds_connection connected from Ok(127.0.0.1:36338)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.103878626Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"io task","client":"0","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.104485852Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"client":"2","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.104551729Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"client":"0","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.104597981Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"client":"1","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.104773688Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.104791005Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"[0]R flush_numbers[0..12]: [64, 64, 71, 70, 48, 71, 63, 63, 54, 68, 61, 71]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.104910712Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"downstairs","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"[0]R generation[0..12]: [35, 35, 39, 38, 26, 39, 34, 34, 30, 36, 33, 39]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.1049221Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"downstairs","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.104933538Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"downstairs","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"[1]R flush_numbers[0..12]: [64, 64, 71, 70, 48, 71, 63, 63, 54, 68, 61, 71]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.104944226Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"downstairs","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"[1]R generation[0..12]: [35, 35, 39, 38, 26, 39, 34, 34, 30, 36, 33, 39]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.104957204Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"downstairs","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.104968001Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"downstairs","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"[2]R flush_numbers[0..12]: [64, 64, 71, 70, 48, 71, 63, 63, 54, 68, 61, 71]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.10497826Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"downstairs","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"[2]R generation[0..12]: [35, 35, 39, 38, 26, 39, 34, 34, 30, 36, 33, 39]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.104988608Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"downstairs","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.104999396Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"downstairs","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"Max found gen is 40","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.105008724Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"downstairs","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"Generation requested: 40 >= found:40","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.105018652Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"downstairs","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"Next flush: 72","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.10502794Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"downstairs","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.105037448Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"":"downstairs","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.105047087Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.105055785Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"client":"0","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.105065423Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"client":"1","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.105074881Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"client":"2","":"downstairs","session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.10508472Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.105093798Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"a886f3b6-219b-4813-bda2-a4085eb69095 is now active with session: e67711fe-a9c1-47f5-9bc6-5689a0bcda16","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.105104646Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.105113284Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"session_id":"e67711fe-a9c1-47f5-9bc6-5689a0bcda16"} {"msg":"The guest has finished waiting for activation with:40","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.105122383Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293} {"msg":"Activated sub_volume a886f3b6-219b-4813-bda2-a4085eb69095","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.105130251Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.10513829Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.10545663Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"task":"crutest"} 01/30 Write block 62 len 4096 data: 6 02/30 Read block 496 len 28672 03/30 Read block 554 len 28672 04/30 Write block 539 len 32768 data: 3 3 3 3 5 4 5 4 05/30 Write block 27 len 32768 data: 4 5 4 4 4 5 4 4 06/30 Read block 199 len 32768 07/30 Read block 596 len 4096 08/30 Read block 102 len 4096 09/30 Read block 412 len 8192 10/30 Write block 162 len 20480 data: 3 4 3 4 4 11/30 Write block 100 len 24576 data: 3 2 2 3 3 3 12/30 Write block 210 len 36864 data: 5 4 5 5 6 5 5 4 4 13/30 Read block 238 len 16384 14/30 Read block 446 len 12288 15/30 Read block 33 len 32768 16/30 Write block 257 len 4096 data: 6 17/30 Read block 76 len 24576 18/30 Flush 19/30 Flush 20/30 Flush 21/30 Write block 488 len 32768 data: 4 6 6 6 5 4 4 5 22/30 Read block 112 len 8192 23/30 Write block 414 len 24576 data: 3 3 3 3 3 3 24/30 Read block 58 len 12288 25/30 Write block 390 len 40960 data: 2 3 3 4 3 3 4 4 4 3 26/30 Write block 76 len 28672 data: 5 4 5 4 4 4 4 27/30 Read block 356 len 40960 28/30 Read block 184 len 28672 29/30 Write block 530 len 36864 data: 4 4 4 4 4 4 3 2 2 30/30 Write block 489 len 24576 data: 7 7 7 6 5 5 ---------------------------------------------------------------- Crucible gen:40 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 7 Done Sent Done false 1002 Acked Read 7 Done Sent Done false 1003 Acked Write 8 Done Sent Done false 1004 Acked Write 8 Done Sent Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Read 1 Done Sent Done false 1007 Acked Read 1 Done Sent Done false 1008 Acked Read 2 Done Sent Done false 1009 Acked Write 5 Done Sent Done false 1010 Acked Write 6 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Read 4 Done Sent Done false 1013 Acked Read 3 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Read 6 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 8 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Read 3 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Read 10 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 29 2 33 Done 28 1 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:43.114303193Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:43.114470842Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1293,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 64 64 64 F F F 1 020-039 40 35 40 72 64 72 F F F <--- 2 040-059 39 39 39 71 71 71 F F F 3 060-079 40 38 40 72 70 72 T T T <--- 4 080-099 26 26 26 48 48 48 T F T <--- 5 100-119 40 39 40 72 71 72 F F F <--- 6 120-139 34 34 34 63 63 63 F F F 7 140-159 34 34 34 63 63 63 F F F 8 160-179 40 30 40 72 54 72 F F F <--- 9 180-199 36 36 36 68 68 68 F F F 10 200-219 40 33 40 72 61 72 F F F <--- 11 220-239 39 39 39 71 71 71 F F F 12 240-259 40 34 40 72 63 72 F F F <--- 13 260-279 36 36 36 65 65 65 F F F 14 280-299 39 39 39 71 71 71 F F F 15 300-319 36 36 36 67 67 67 F F F 16 320-339 37 37 37 69 69 69 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 29 29 29 52 52 52 T F T <--- 20 400-419 28 28 28 51 51 51 T F T <--- 21 420-439 37 37 37 69 69 69 F F F 22 440-459 37 37 37 69 69 69 F F F 23 460-479 39 39 39 71 71 71 F F F 24 480-499 36 36 36 67 67 67 T F T <--- 25 500-519 37 37 37 69 69 69 F F F 26 520-539 40 37 40 72 69 72 T F T <--- 27 540-559 40 38 40 72 70 72 F F F <--- 28 560-579 37 37 37 69 69 69 F F F 29 580-599 36 36 36 65 65 65 F F F Max gen: 40, Max flush: 72 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 20 Verifying data now No change after verify Loop: 20 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 64 64 64 F F F 1 020-039 40 40 40 72 72 72 F F F 2 040-059 39 39 39 71 71 71 F F F 3 060-079 41 41 41 73 73 73 F F F 4 080-099 41 41 41 73 73 73 F F F 5 100-119 40 40 40 72 72 72 F F F 6 120-139 34 34 34 63 63 63 F F F 7 140-159 34 34 34 63 63 63 F F F 8 160-179 40 40 40 72 72 72 F F F 9 180-199 36 36 36 68 68 68 F F F 10 200-219 40 40 40 72 72 72 F F F 11 220-239 39 39 39 71 71 71 F F F 12 240-259 40 40 40 72 72 72 F F F 13 260-279 36 36 36 65 65 65 F F F 14 280-299 39 39 39 71 71 71 F F F 15 300-319 36 36 36 67 67 67 F F F 16 320-339 37 37 37 69 69 69 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 41 41 41 73 73 73 F F F 20 400-419 41 41 41 73 73 73 F F F 21 420-439 37 37 37 69 69 69 F F F 22 440-459 37 37 37 69 69 69 F F F 23 460-479 39 39 39 71 71 71 F F F 24 480-499 41 41 41 73 73 73 F F F 25 500-519 37 37 37 69 69 69 F F F 26 520-539 41 41 41 73 73 73 F F F 27 540-559 40 40 40 72 72 72 F F F 28 560-579 37 37 37 69 69 69 F F F 29 580-599 36 36 36 65 65 65 F F F Max gen: 41, Max flush: 73 Begin loop 21 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.360098882Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.368254345Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.368577004Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.368606449Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.368615017Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.368626425Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301} {"msg":"Crucible 58e2be96-a6d9-4c0e-9598-16bfe2dfa86e has session id: 8d89901f-54c3-447a-b92c-a84dc6960338","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.368982828Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"Upstairs opts: Upstairs UUID: 58e2be96-a6d9-4c0e-9598-16bfe2dfa86e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369024531Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"Crucible stats registered with UUID: 58e2be96-a6d9-4c0e-9598-16bfe2dfa86e","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369037908Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:47.369047206Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369176642Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"io task","client":"1","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369207696Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"io task","client":"0","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369218794Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"io task","client":"2","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369367066Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"task":"crutest"} {"msg":"The guest has requested activation with gen:42","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369382594Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301} {"msg":"58e2be96-a6d9-4c0e-9598-16bfe2dfa86e active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369414448Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369423896Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"client":"0","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369433684Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"client":"1","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369443262Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"client":"2","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.36945283Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"io task","client":"2","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369462859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"io task","client":"0","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369472707Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"io task","client":"1","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"ds_connection connected from Ok(127.0.0.1:37118)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.36951077Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"io task","client":"0","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"ds_connection connected from Ok(127.0.0.1:65308)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369527776Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"io task","client":"2","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"ds_connection connected from Ok(127.0.0.1:65465)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.369538334Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"io task","client":"1","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.37031088Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"client":"2","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370334845Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"client":"1","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370352522Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"client":"0","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370645887Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370669593Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"[0]R flush_numbers[0..12]: [64, 72, 71, 73, 73, 72, 63, 63, 72, 68, 72, 71]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.37068033Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"downstairs","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"[0]R generation[0..12]: [35, 40, 39, 41, 41, 40, 34, 34, 40, 36, 40, 39]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370691348Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"downstairs","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370702466Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"downstairs","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"[1]R flush_numbers[0..12]: [64, 72, 71, 73, 73, 72, 63, 63, 72, 68, 72, 71]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370712894Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"downstairs","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"[1]R generation[0..12]: [35, 40, 39, 41, 41, 40, 34, 34, 40, 36, 40, 39]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370723312Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"downstairs","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.37073428Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"downstairs","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"[2]R flush_numbers[0..12]: [64, 72, 71, 73, 73, 72, 63, 63, 72, 68, 72, 71]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370744728Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"downstairs","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"[2]R generation[0..12]: [35, 40, 39, 41, 41, 40, 34, 34, 40, 36, 40, 39]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370755166Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"downstairs","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370765964Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"downstairs","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"Max found gen is 42","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370777592Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"downstairs","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"Generation requested: 42 >= found:42","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.37078722Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"downstairs","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"Next flush: 74","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370796439Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"downstairs","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370805767Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"":"downstairs","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370815225Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370823794Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"client":"0","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370833472Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"client":"1","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.37084284Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"client":"2","":"downstairs","session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370852468Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370861487Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"58e2be96-a6d9-4c0e-9598-16bfe2dfa86e is now active with session: 8d89901f-54c3-447a-b92c-a84dc6960338","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370871255Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370879923Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"session_id":"8d89901f-54c3-447a-b92c-a84dc6960338"} {"msg":"The guest has finished waiting for activation with:42","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370888801Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301} {"msg":"Activated sub_volume 58e2be96-a6d9-4c0e-9598-16bfe2dfa86e","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.37089667Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.370904638Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.371212131Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"task":"crutest"} 01/30 Write block 544 len 4096 data: 5 02/30 Write block 456 len 36864 data: 3 2 2 2 2 2 2 2 3 03/30 Read block 192 len 8192 04/30 Write block 107 len 20480 data: 4 4 5 5 5 05/30 Read block 279 len 4096 06/30 Read block 149 len 16384 07/30 Write block 42 len 8192 data: 5 5 08/30 Read block 455 len 4096 09/30 Write block 276 len 24576 data: 4 4 3 4 3 2 10/30 Read block 120 len 16384 11/30 Read block 288 len 36864 12/30 Write block 137 len 36864 data: 4 5 5 5 5 5 4 5 5 13/30 Write block 274 len 28672 data: 4 5 5 5 4 5 4 14/30 Write block 249 len 16384 data: 4 3 4 5 15/30 Read block 361 len 24576 16/30 Write block 420 len 24576 data: 2 2 2 2 2 2 17/30 Write block 83 len 4096 data: 2 18/30 Read block 195 len 8192 19/30 Read block 556 len 16384 20/30 Read block 350 len 8192 21/30 Write block 307 len 20480 data: 10 9 9 10 8 22/30 Read block 517 len 4096 23/30 Read block 198 len 16384 24/30 Write block 217 len 36864 data: 5 5 4 4 5 4 3 4 4 25/30 Read block 16 len 28672 26/30 Read block 156 len 40960 27/30 Read block 140 len 24576 28/30 Read block 48 len 20480 29/30 Write block 575 len 40960 data: 3 3 3 4 3 3 3 4 3 2 30/30 Read block 493 len 32768 ---------------------------------------------------------------- Crucible gen:42 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Sent Done false 1001 Acked Write 9 Done Sent Done false 1002 Acked Read 2 Done Sent Done false 1003 Acked Write 5 Done Sent Done false 1004 Acked Read 1 Done Sent Done false 1005 Acked Read 4 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Read 1 Done Sent Done false 1008 Acked Write 6 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Read 9 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Write 7 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Write 1 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Read 2 Done Sent Done false 1020 Acked Write 5 Done Sent Done false 1021 Acked Read 1 Done Sent Done false 1022 Acked Read 4 Done Sent Done false 1023 Acked Write 9 Done Sent Done false 1024 Acked Read 7 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Read 6 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Write 10 Done Sent Done false 1029 Acked Read 8 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:47.379838055Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:47.380011603Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1301,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 64 64 64 F F F 1 020-039 40 40 40 72 72 72 F F F 2 040-059 39 39 39 71 71 71 T F T <--- 3 060-079 41 41 41 73 73 73 F F F 4 080-099 41 41 41 73 73 73 T F T <--- 5 100-119 40 40 40 72 72 72 T F T <--- 6 120-139 34 34 34 63 63 63 T F T <--- 7 140-159 34 34 34 63 63 63 T F T <--- 8 160-179 40 40 40 72 72 72 F F F 9 180-199 36 36 36 68 68 68 F F F 10 200-219 40 40 40 72 72 72 T F T <--- 11 220-239 39 39 39 71 71 71 T F T <--- 12 240-259 40 40 40 72 72 72 T F T <--- 13 260-279 36 36 36 65 65 65 T F T <--- 14 280-299 39 39 39 71 71 71 T F T <--- 15 300-319 36 36 36 67 67 67 T F T <--- 16 320-339 37 37 37 69 69 69 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 41 41 41 73 73 73 F F F 20 400-419 41 41 41 73 73 73 F F F 21 420-439 37 37 37 69 69 69 T F T <--- 22 440-459 37 37 37 69 69 69 T F T <--- 23 460-479 39 39 39 71 71 71 T F T <--- 24 480-499 41 41 41 73 73 73 F F F 25 500-519 37 37 37 69 69 69 F F F 26 520-539 41 41 41 73 73 73 F F F 27 540-559 40 40 40 72 72 72 T F T <--- 28 560-579 37 37 37 69 69 69 T F T <--- 29 580-599 36 36 36 65 65 65 T F T <--- Max gen: 41, Max flush: 73 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 21 Verifying data now No change after verify Loop: 21 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 64 64 64 F F F 1 020-039 40 40 40 72 72 72 F F F 2 040-059 42 42 42 74 74 74 F F F 3 060-079 41 41 41 73 73 73 F F F 4 080-099 42 42 42 74 74 74 F F F 5 100-119 42 42 42 74 74 74 F F F 6 120-139 42 42 42 74 74 74 F F F 7 140-159 42 42 42 74 74 74 F F F 8 160-179 40 40 40 72 72 72 F F F 9 180-199 36 36 36 68 68 68 F F F 10 200-219 42 42 42 74 74 74 F F F 11 220-239 42 42 42 74 74 74 F F F 12 240-259 42 42 42 74 74 74 F F F 13 260-279 42 42 42 74 74 74 F F F 14 280-299 42 42 42 74 74 74 F F F 15 300-319 42 42 42 74 74 74 F F F 16 320-339 37 37 37 69 69 69 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 27 27 27 49 49 49 F F F 19 380-399 41 41 41 73 73 73 F F F 20 400-419 41 41 41 73 73 73 F F F 21 420-439 42 42 42 74 74 74 F F F 22 440-459 42 42 42 74 74 74 F F F 23 460-479 42 42 42 74 74 74 F F F 24 480-499 41 41 41 73 73 73 F F F 25 500-519 37 37 37 69 69 69 F F F 26 520-539 41 41 41 73 73 73 F F F 27 540-559 42 42 42 74 74 74 F F F 28 560-579 42 42 42 74 74 74 F F F 29 580-599 42 42 42 74 74 74 F F F Max gen: 42, Max flush: 74 Begin loop 22 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.643551349Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.651638604Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.651938478Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.651967923Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.651976381Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.651987739Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309} {"msg":"Crucible 5ae30b3e-89b8-4249-bf44-95c7d1ad616d has session id: c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652321046Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"Upstairs opts: Upstairs UUID: 5ae30b3e-89b8-4249-bf44-95c7d1ad616d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652345112Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"Crucible stats registered with UUID: 5ae30b3e-89b8-4249-bf44-95c7d1ad616d","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.65235468Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:51.652363888Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652520279Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"io task","client":"0","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652553143Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"io task","client":"2","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652564181Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"io task","client":"1","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652707454Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"task":"crutest"} {"msg":"The guest has requested activation with gen:44","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.65272633Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309} {"msg":"5ae30b3e-89b8-4249-bf44-95c7d1ad616d active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652752236Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652761724Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"client":"0","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652771282Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"client":"1","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.65278075Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"client":"2","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652790288Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"io task","client":"2","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652825132Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"io task","client":"0","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652848418Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"io task","client":"1","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"ds_connection connected from Ok(127.0.0.1:37379)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652876422Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"io task","client":"1","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"ds_connection connected from Ok(127.0.0.1:49536)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.65288785Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"io task","client":"2","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"ds_connection connected from Ok(127.0.0.1:56712)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.652899048Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"io task","client":"0","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.653593328Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"client":"2","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.653617204Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"client":"1","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.653748829Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"client":"0","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.65395782Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.653993863Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"[0]R flush_numbers[0..12]: [64, 72, 74, 73, 74, 74, 74, 74, 72, 68, 74, 74]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654046273Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"downstairs","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"[0]R generation[0..12]: [35, 40, 42, 41, 42, 42, 42, 42, 40, 36, 42, 42]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654057831Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"downstairs","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654068899Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"downstairs","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"[1]R flush_numbers[0..12]: [64, 72, 74, 73, 74, 74, 74, 74, 72, 68, 74, 74]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654079297Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"downstairs","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"[1]R generation[0..12]: [35, 40, 42, 41, 42, 42, 42, 42, 40, 36, 42, 42]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654089545Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"downstairs","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654100383Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"downstairs","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"[2]R flush_numbers[0..12]: [64, 72, 74, 73, 74, 74, 74, 74, 72, 68, 74, 74]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654110641Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"downstairs","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"[2]R generation[0..12]: [35, 40, 42, 41, 42, 42, 42, 42, 40, 36, 42, 42]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654120789Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"downstairs","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654131557Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"downstairs","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"Max found gen is 43","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654140986Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"downstairs","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"Generation requested: 44 >= found:43","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654150624Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"downstairs","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"Next flush: 75","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654159952Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"downstairs","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.65416937Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"":"downstairs","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654178978Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654188997Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"client":"0","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654198525Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"client":"1","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654208073Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"client":"2","":"downstairs","session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654217851Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654226919Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"5ae30b3e-89b8-4249-bf44-95c7d1ad616d is now active with session: c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654236658Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654245356Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"session_id":"c1fe6aa3-7e0d-4d5e-9ecd-913c2ec5bb13"} {"msg":"The guest has finished waiting for activation with:44","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654254384Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309} {"msg":"Activated sub_volume 5ae30b3e-89b8-4249-bf44-95c7d1ad616d","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654262163Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654270051Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.654585632Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"task":"crutest"} 01/30 Write block 218 len 12288 data: 6 5 5 02/30 Read block 34 len 4096 03/30 Read block 271 len 8192 04/30 Read block 232 len 32768 05/30 Read block 183 len 24576 06/30 Read block 222 len 28672 07/30 Read block 555 len 8192 08/30 Read block 216 len 4096 09/30 Read block 583 len 16384 10/30 Flush 11/30 Write block 34 len 36864 data: 5 4 3 3 3 3 3 4 6 12/30 Read block 235 len 8192 13/30 Write block 578 len 28672 data: 5 4 4 4 5 4 3 14/30 Write block 311 len 28672 data: 9 7 7 7 5 5 6 15/30 Write block 364 len 12288 data: 3 3 2 16/30 Write block 585 len 40960 data: 2 4 6 7 7 7 6 7 8 8 17/30 Read block 116 len 40960 18/30 Write block 396 len 8192 data: 5 5 19/30 Read block 590 len 20480 20/30 Read block 479 len 24576 21/30 Read block 432 len 4096 22/30 Read block 511 len 40960 23/30 Flush 24/30 Write block 82 len 4096 data: 5 25/30 Read block 336 len 24576 26/30 Write block 524 len 24576 data: 5 5 4 5 5 5 27/30 Read block 229 len 32768 28/30 Read block 379 len 24576 29/30 Write block 59 len 16384 data: 6 6 5 7 30/30 Read block 81 len 8192 ---------------------------------------------------------------- Crucible gen:44 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 1 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Read 6 Done Sent Done false 1005 Acked Read 7 Done Sent Done false 1006 Acked Read 2 Done Sent Done false 1007 Acked Read 1 Done Sent Done false 1008 Acked Read 4 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 9 Done Sent Done false 1011 Acked Read 2 Done Sent Done false 1012 Acked Write 7 Done Sent Done false 1013 Acked Write 7 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Read 10 Done Sent Done false 1017 Acked Write 2 Done Sent Done false 1018 Acked Read 5 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Read 1 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Read 6 Done Sent Done false 1025 Acked Write 6 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Write 4 Done Sent Done false 1029 Acked Read 2 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 26 0 26 Done 30 4 30 64 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1022)) None Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:51.663539455Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:51.663711073Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1309,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 64 64 64 F F F 1 020-039 44 40 44 76 72 76 F F F <--- 2 040-059 44 42 44 76 74 76 T F T <--- 3 060-079 41 41 41 73 73 73 T F T <--- 4 080-099 42 42 42 74 74 74 T F T <--- 5 100-119 42 42 42 74 74 74 F F F 6 120-139 42 42 42 74 74 74 F F F 7 140-159 42 42 42 74 74 74 F F F 8 160-179 40 40 40 72 72 72 F F F 9 180-199 36 36 36 68 68 68 F F F 10 200-219 44 42 44 75 74 75 F T F <--- 11 220-239 44 42 44 75 74 75 F T F <--- 12 240-259 42 42 42 74 74 74 F F F 13 260-279 42 42 42 74 74 74 F F F 14 280-299 42 42 42 74 74 74 F F F 15 300-319 44 42 44 76 74 76 F F F <--- 16 320-339 37 37 37 69 69 69 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 44 27 44 76 49 76 F F F <--- 19 380-399 44 41 44 76 73 76 F F F <--- 20 400-419 41 41 41 73 73 73 F F F 21 420-439 42 42 42 74 74 74 F F F 22 440-459 42 42 42 74 74 74 F F F 23 460-479 42 42 42 74 74 74 F F F 24 480-499 41 41 41 73 73 73 F F F 25 500-519 37 37 37 69 69 69 F F F 26 520-539 41 41 41 73 73 73 T F T <--- 27 540-559 42 42 42 74 74 74 F F F 28 560-579 44 42 44 76 74 76 F F F <--- 29 580-599 44 42 44 76 74 76 F F F <--- Max gen: 44, Max flush: 76 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 22 Verifying data now No change after verify Loop: 22 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 64 64 64 F F F 1 020-039 44 44 44 76 76 76 F F F 2 040-059 45 45 45 77 77 77 F F F 3 060-079 45 45 45 77 77 77 F F F 4 080-099 45 45 45 77 77 77 F F F 5 100-119 42 42 42 74 74 74 F F F 6 120-139 42 42 42 74 74 74 F F F 7 140-159 42 42 42 74 74 74 F F F 8 160-179 40 40 40 72 72 72 F F F 9 180-199 36 36 36 68 68 68 F F F 10 200-219 44 44 44 75 75 75 F F F 11 220-239 44 44 44 75 75 75 F F F 12 240-259 42 42 42 74 74 74 F F F 13 260-279 42 42 42 74 74 74 F F F 14 280-299 42 42 42 74 74 74 F F F 15 300-319 44 44 44 76 76 76 F F F 16 320-339 37 37 37 69 69 69 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 44 44 44 76 76 76 F F F 19 380-399 44 44 44 76 76 76 F F F 20 400-419 41 41 41 73 73 73 F F F 21 420-439 42 42 42 74 74 74 F F F 22 440-459 42 42 42 74 74 74 F F F 23 460-479 42 42 42 74 74 74 F F F 24 480-499 41 41 41 73 73 73 F F F 25 500-519 37 37 37 69 69 69 F F F 26 520-539 45 45 45 77 77 77 F F F 27 540-559 42 42 42 74 74 74 F F F 28 560-579 44 44 44 76 76 76 F F F 29 580-599 44 44 44 76 76 76 F F F Max gen: 45, Max flush: 77 Begin loop 23 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.909106997Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:55.91624432Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.91629843Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.923169163Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.923437193Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.923462868Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.923474176Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.923485664Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317} {"msg":"Crucible 6db42ff4-556d-421e-afba-872826613e03 has session id: d22b9469-2487-4217-a660-eda281c465c8","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.923849636Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"Upstairs opts: Upstairs UUID: 6db42ff4-556d-421e-afba-872826613e03, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.92387851Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"Crucible stats registered with UUID: 6db42ff4-556d-421e-afba-872826613e03","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.923889818Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:55.923899166Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.92403957Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"io task","client":"1","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.924070154Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"io task","client":"0","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.924081502Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"io task","client":"2","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.924218627Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"task":"crutest"} {"msg":"The guest has requested activation with gen:46","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.924230414Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317} {"msg":"6db42ff4-556d-421e-afba-872826613e03 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.924248451Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.924257349Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"client":"0","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.924267127Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"client":"1","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.924276806Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"client":"2","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.924288213Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"io task","client":"2","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.924298272Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"io task","client":"0","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.92430811Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"io task","client":"1","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"ds_connection connected from Ok(127.0.0.1:61863)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.924364599Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"io task","client":"1","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"ds_connection connected from Ok(127.0.0.1:59034)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.924375707Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"io task","client":"0","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"ds_connection connected from Ok(127.0.0.1:37778)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.924463411Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"io task","client":"2","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.92515883Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"client":"1","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925184866Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"client":"2","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925196903Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"client":"0","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925476471Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925504496Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"[0]R flush_numbers[0..12]: [64, 76, 77, 77, 77, 74, 74, 74, 72, 68, 75, 75]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925585411Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"downstairs","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"[0]R generation[0..12]: [35, 44, 45, 45, 45, 42, 42, 42, 40, 36, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925596968Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"downstairs","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925607826Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"downstairs","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"[1]R flush_numbers[0..12]: [64, 76, 77, 77, 77, 74, 74, 74, 72, 68, 75, 75]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925618174Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"downstairs","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"[1]R generation[0..12]: [35, 44, 45, 45, 45, 42, 42, 42, 40, 36, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925631212Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"downstairs","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.92564209Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"downstairs","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"[2]R flush_numbers[0..12]: [64, 76, 77, 77, 77, 74, 74, 74, 72, 68, 75, 75]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925652688Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"downstairs","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"[2]R generation[0..12]: [35, 44, 45, 45, 45, 42, 42, 42, 40, 36, 44, 44]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925662966Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"downstairs","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925673654Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"downstairs","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"Max found gen is 46","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925682992Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"downstairs","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"Generation requested: 46 >= found:46","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.9256926Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"downstairs","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"Next flush: 78","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925702079Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"downstairs","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925711497Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"":"downstairs","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925721255Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925729933Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"client":"0","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925739702Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"client":"1","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.92574955Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"client":"2","":"downstairs","session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925759388Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925768346Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"6db42ff4-556d-421e-afba-872826613e03 is now active with session: d22b9469-2487-4217-a660-eda281c465c8","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925779624Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925788263Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"session_id":"d22b9469-2487-4217-a660-eda281c465c8"} {"msg":"The guest has finished waiting for activation with:46","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925797231Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317} {"msg":"Activated sub_volume 6db42ff4-556d-421e-afba-872826613e03","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925805119Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.925813138Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.926149135Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"task":"crutest"} 01/30 Write block 182 len 20480 data: 5 6 6 5 6 02/30 Write block 464 len 32768 data: 4 3 3 4 4 4 4 4 03/30 Write block 271 len 4096 data: 4 04/30 Flush 05/30 Write block 215 len 16384 data: 6 6 6 7 06/30 Read block 484 len 28672 07/30 Write block 379 len 8192 data: 6 7 08/30 Write block 82 len 36864 data: 6 3 5 6 6 4 4 4 4 09/30 Flush 10/30 Write block 37 len 40960 data: 4 4 4 4 5 7 6 5 5 5 11/30 Read block 31 len 36864 12/30 Write block 484 len 16384 data: 5 5 6 6 13/30 Write block 439 len 20480 data: 2 4 4 4 5 14/30 Read block 307 len 20480 15/30 Read block 226 len 12288 16/30 Read block 198 len 8192 17/30 Read block 434 len 8192 18/30 Read block 77 len 12288 19/30 Flush 20/30 Flush 21/30 Write block 138 len 24576 data: 6 6 6 6 6 5 22/30 Read block 558 len 12288 23/30 Read block 317 len 8192 24/30 Read block 245 len 20480 25/30 Read block 561 len 20480 26/30 Write block 75 len 4096 data: 4 27/30 Read block 167 len 40960 28/30 Write block 470 len 40960 data: 5 5 4 3 2 2 2 3 3 4 29/30 Write block 568 len 16384 data: 4 4 4 4 30/30 Write block 303 len 8192 data: 6 8 ---------------------------------------------------------------- Crucible gen:46 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 8 Sent Done Done false 1002 Acked Write 1 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 4 Sent Done Done false 1005 Acked Read 7 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Write 9 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Write 4 Sent Done Done false 1012 Acked Write 5 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Read 3 Sent Done Done false 1015 Acked Read 2 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Flush 0 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Read 3 Sent Done Done false 1022 Acked Read 2 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 4 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 3 3 35 Done 1 27 27 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:57:55.935009946Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:57:55.935167056Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1317,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 64 64 64 F F F 1 020-039 44 46 46 76 80 80 F F F <--- 2 040-059 45 46 46 77 80 80 F F F <--- 3 060-079 45 45 45 77 77 77 F T T <--- 4 080-099 45 46 46 77 79 79 F F F <--- 5 100-119 42 42 42 74 74 74 F F F 6 120-139 42 42 42 74 74 74 F T T <--- 7 140-159 42 42 42 74 74 74 F T T <--- 8 160-179 40 40 40 72 72 72 F F F 9 180-199 36 46 46 68 78 78 T F F <--- 10 200-219 44 46 46 75 79 79 F F F <--- 11 220-239 44 44 44 75 75 75 F F F 12 240-259 42 42 42 74 74 74 F F F 13 260-279 42 46 46 74 78 78 F F F <--- 14 280-299 42 42 42 74 74 74 F F F 15 300-319 44 44 44 76 76 76 F T T <--- 16 320-339 37 37 37 69 69 69 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 44 46 46 76 79 79 F F F <--- 19 380-399 44 46 46 76 79 79 F F F <--- 20 400-419 41 41 41 73 73 73 F F F 21 420-439 42 46 46 74 80 80 F F F <--- 22 440-459 42 46 46 74 80 80 F F F <--- 23 460-479 42 46 46 74 78 78 F T T <--- 24 480-499 41 46 46 73 80 80 F F F <--- 25 500-519 37 37 37 69 69 69 F F F 26 520-539 45 45 45 77 77 77 F F F 27 540-559 42 42 42 74 74 74 F F F 28 560-579 44 44 44 76 76 76 F T T <--- 29 580-599 44 44 44 76 76 76 F F F Max gen: 46, Max flush: 80 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 23 Verifying data now No change after verify Loop: 23 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 35 35 35 64 64 64 F F F 1 020-039 46 46 46 80 80 80 F F F 2 040-059 46 46 46 80 80 80 F F F 3 060-079 47 47 47 81 81 81 F F F 4 080-099 46 46 46 79 79 79 F F F 5 100-119 42 42 42 74 74 74 F F F 6 120-139 47 47 47 81 81 81 F F F 7 140-159 47 47 47 81 81 81 F F F 8 160-179 40 40 40 72 72 72 F F F 9 180-199 46 46 46 78 78 78 F F F 10 200-219 46 46 46 79 79 79 F F F 11 220-239 44 44 44 75 75 75 F F F 12 240-259 42 42 42 74 74 74 F F F 13 260-279 46 46 46 78 78 78 F F F 14 280-299 42 42 42 74 74 74 F F F 15 300-319 47 47 47 81 81 81 F F F 16 320-339 37 37 37 69 69 69 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 46 46 46 79 79 79 F F F 19 380-399 46 46 46 79 79 79 F F F 20 400-419 41 41 41 73 73 73 F F F 21 420-439 46 46 46 80 80 80 F F F 22 440-459 46 46 46 80 80 80 F F F 23 460-479 47 47 47 81 81 81 F F F 24 480-499 46 46 46 80 80 80 F F F 25 500-519 37 37 37 69 69 69 F F F 26 520-539 45 45 45 77 77 77 F F F 27 540-559 42 42 42 74 74 74 F F F 28 560-579 47 47 47 81 81 81 F F F 29 580-599 44 44 44 76 76 76 F F F Max gen: 47, Max flush: 81 Begin loop 24 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.202236641Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.210539356Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.210846448Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.210878023Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.210886481Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.210897809Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325} {"msg":"Crucible 8a90de3a-6319-4be6-be69-cbf2a8954667 has session id: 66c229da-edaf-4aee-854d-ecda690b5c92","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211234616Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"Upstairs opts: Upstairs UUID: 8a90de3a-6319-4be6-be69-cbf2a8954667, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.21126426Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"Crucible stats registered with UUID: 8a90de3a-6319-4be6-be69-cbf2a8954667","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211278068Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:00.211287336Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211412952Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"io task","client":"1","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211430619Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"io task","client":"0","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211440627Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"io task","client":"2","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211610085Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"task":"crutest"} {"msg":"The guest has requested activation with gen:48","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211628192Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325} {"msg":"8a90de3a-6319-4be6-be69-cbf2a8954667 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211663075Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211672624Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"client":"0","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211682182Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"client":"1","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.21169177Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"client":"2","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211701268Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"io task","client":"2","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211711136Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"io task","client":"0","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211724934Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"io task","client":"1","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"ds_connection connected from Ok(127.0.0.1:49580)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211776214Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"io task","client":"2","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"ds_connection connected from Ok(127.0.0.1:47689)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.2118012Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"io task","client":"0","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"ds_connection connected from Ok(127.0.0.1:33967)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.211812717Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"io task","client":"1","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212458447Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"client":"0","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212470384Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"client":"1","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212487751Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"client":"2","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.2127607Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212770278Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"[0]R flush_numbers[0..12]: [64, 80, 80, 81, 79, 74, 81, 81, 72, 78, 79, 75]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212847924Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"downstairs","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"[0]R generation[0..12]: [35, 46, 46, 47, 46, 42, 47, 47, 40, 46, 46, 44]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212858912Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"downstairs","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212869959Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"downstairs","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"[1]R flush_numbers[0..12]: [64, 80, 80, 81, 79, 74, 81, 81, 72, 78, 79, 75]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212880208Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"downstairs","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"[1]R generation[0..12]: [35, 46, 46, 47, 46, 42, 47, 47, 40, 46, 46, 44]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212890526Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"downstairs","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212901134Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"downstairs","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"[2]R flush_numbers[0..12]: [64, 80, 80, 81, 79, 74, 81, 81, 72, 78, 79, 75]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212911442Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"downstairs","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"[2]R generation[0..12]: [35, 46, 46, 47, 46, 42, 47, 47, 40, 46, 46, 44]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.21292168Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"downstairs","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212932808Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"downstairs","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"Max found gen is 48","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212943536Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"downstairs","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"Generation requested: 48 >= found:48","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212953264Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"downstairs","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"Next flush: 82","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212962482Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"downstairs","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.21297177Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"":"downstairs","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212981329Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212990007Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"client":"0","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.212999635Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"client":"1","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.213009203Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"client":"2","":"downstairs","session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.213019132Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.2130282Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"8a90de3a-6319-4be6-be69-cbf2a8954667 is now active with session: 66c229da-edaf-4aee-854d-ecda690b5c92","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.213038038Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.213046786Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"session_id":"66c229da-edaf-4aee-854d-ecda690b5c92"} {"msg":"The guest has finished waiting for activation with:48","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.213055855Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325} {"msg":"Activated sub_volume 8a90de3a-6319-4be6-be69-cbf2a8954667","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.213063793Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.213071832Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.213344391Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"task":"crutest"} 01/30 Flush 02/30 Write block 210 len 12288 data: 6 5 6 03/30 Read block 487 len 16384 04/30 Read block 392 len 24576 05/30 Write block 202 len 36864 data: 2 2 3 3 5 5 6 7 7 06/30 Write block 93 len 8192 data: 4 5 07/30 Read block 547 len 40960 08/30 Write block 497 len 40960 data: 4 5 5 5 5 5 6 6 7 6 09/30 Read block 86 len 32768 10/30 Read block 62 len 32768 11/30 Read block 303 len 8192 12/30 Read block 535 len 20480 13/30 Flush 14/30 Write block 559 len 8192 data: 8 7 15/30 Read block 380 len 40960 16/30 Write block 392 len 32768 data: 4 5 4 4 6 6 5 4 17/30 Write block 558 len 28672 data: 7 9 8 7 7 7 5 18/30 Read block 350 len 4096 19/30 Flush 20/30 Write block 15 len 24576 data: 4 5 4 5 5 6 21/30 Read block 471 len 20480 22/30 Read block 120 len 12288 23/30 Read block 315 len 32768 24/30 Read block 182 len 28672 25/30 Flush 26/30 Write block 435 len 16384 data: 2 2 2 2 27/30 Write block 18 len 16384 data: 6 6 7 5 28/30 Read block 229 len 12288 29/30 Write block 36 len 28672 data: 4 5 5 5 5 6 8 30/30 Write block 314 len 32768 data: 8 6 6 7 4 5 6 6 ---------------------------------------------------------------- Crucible gen:48 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 3 Done Done Done false 1002 Acked Read 4 Done Done Done false 1003 Acked Read 6 Done Done Done false 1004 Acked Write 9 Done Done Sent false 1005 Acked Write 2 Done Done Sent false 1006 Acked Read 10 Done Done Sent false 1007 Acked Write 10 Done Done Sent false 1008 Acked Read 8 Done Done Sent false 1009 Acked Read 8 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Read 5 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Read 10 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Read 1 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 6 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Read 8 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Read 3 Done Done Sent false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 26 30 Done 27 27 3 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:00.222137754Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:00.222325189Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1325,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 48 48 35 85 85 64 T T F <--- 1 020-039 48 48 46 85 85 80 T T F <--- 2 040-059 46 46 46 80 80 80 T T F <--- 3 060-079 47 47 47 81 81 81 F F F 4 080-099 48 48 46 83 83 79 F F F <--- 5 100-119 42 42 42 74 74 74 F F F 6 120-139 47 47 47 81 81 81 F F F 7 140-159 47 47 47 81 81 81 F F F 8 160-179 40 40 40 72 72 72 F F F 9 180-199 46 46 46 78 78 78 F F F 10 200-219 48 48 46 83 83 79 F F T <--- 11 220-239 44 44 44 75 75 75 F F F 12 240-259 42 42 42 74 74 74 F F F 13 260-279 46 46 46 78 78 78 F F F 14 280-299 42 42 42 74 74 74 F F F 15 300-319 47 47 47 81 81 81 T T F <--- 16 320-339 37 37 37 69 69 69 T T F <--- 17 340-359 35 35 35 64 64 64 F F F 18 360-379 46 46 46 79 79 79 F F F 19 380-399 48 48 46 84 84 79 F F F <--- 20 400-419 41 41 41 73 73 73 F F F 21 420-439 46 46 46 80 80 80 T T F <--- 22 440-459 46 46 46 80 80 80 F F F 23 460-479 47 47 47 81 81 81 F F F 24 480-499 48 48 46 83 83 80 F F F <--- 25 500-519 48 48 37 83 83 69 F F F <--- 26 520-539 45 45 45 77 77 77 F F F 27 540-559 48 48 42 84 84 74 F F F <--- 28 560-579 48 48 47 84 84 81 F F F <--- 29 580-599 44 44 44 76 76 76 F F F Max gen: 48, Max flush: 85 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 24 Verifying data now No change after verify Loop: 24 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 49 49 49 86 86 86 F F F 1 020-039 49 49 49 86 86 86 F F F 2 040-059 49 49 49 86 86 86 F F F 3 060-079 47 47 47 81 81 81 F F F 4 080-099 48 48 48 83 83 83 F F F 5 100-119 42 42 42 74 74 74 F F F 6 120-139 47 47 47 81 81 81 F F F 7 140-159 47 47 47 81 81 81 F F F 8 160-179 40 40 40 72 72 72 F F F 9 180-199 46 46 46 78 78 78 F F F 10 200-219 48 48 48 83 83 83 F F F 11 220-239 44 44 44 75 75 75 F F F 12 240-259 42 42 42 74 74 74 F F F 13 260-279 46 46 46 78 78 78 F F F 14 280-299 42 42 42 74 74 74 F F F 15 300-319 49 49 49 86 86 86 F F F 16 320-339 49 49 49 86 86 86 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 46 46 46 79 79 79 F F F 19 380-399 48 48 48 84 84 84 F F F 20 400-419 41 41 41 73 73 73 F F F 21 420-439 49 49 49 86 86 86 F F F 22 440-459 46 46 46 80 80 80 F F F 23 460-479 47 47 47 81 81 81 F F F 24 480-499 48 48 48 83 83 83 F F F 25 500-519 48 48 48 83 83 83 F F F 26 520-539 45 45 45 77 77 77 F F F 27 540-559 48 48 48 84 84 84 F F F 28 560-579 48 48 48 84 84 84 F F F 29 580-599 44 44 44 76 76 76 F F F Max gen: 49, Max flush: 86 Begin loop 25 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.469852663Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.477983271Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.478266508Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.478296062Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.478304341Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.478315678Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333} {"msg":"Crucible f4f18a9f-12b3-44e1-ab7c-2b27467f3313 has session id: a4b5e210-912e-454d-8a2b-02ee09eb9286","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.47867821Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"Upstairs opts: Upstairs UUID: f4f18a9f-12b3-44e1-ab7c-2b27467f3313, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.478724012Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"Crucible stats registered with UUID: f4f18a9f-12b3-44e1-ab7c-2b27467f3313","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.47873454Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:04.478743548Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.478888311Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"io task","client":"0","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.478905238Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"io task","client":"1","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.478915146Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"io task","client":"2","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.47905635Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"task":"crutest"} {"msg":"The guest has requested activation with gen:50","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.479080015Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333} {"msg":"f4f18a9f-12b3-44e1-ab7c-2b27467f3313 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.479098432Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.47910724Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"client":"0","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.479116628Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"client":"1","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.479126097Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"client":"2","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.479135415Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"io task","client":"2","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.479145213Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"io task","client":"0","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.479155151Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"io task","client":"1","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"ds_connection connected from Ok(127.0.0.1:53491)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.47921316Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"io task","client":"1","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"ds_connection connected from Ok(127.0.0.1:58761)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.479224078Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"io task","client":"2","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"ds_connection connected from Ok(127.0.0.1:48088)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.479234316Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"io task","client":"0","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.479924887Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"client":"2","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480010361Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"client":"1","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480058922Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"client":"0","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480273482Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480306466Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"[0]R flush_numbers[0..12]: [86, 86, 86, 81, 83, 74, 81, 81, 72, 78, 83, 75]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480381891Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"downstairs","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"[0]R generation[0..12]: [49, 49, 49, 47, 48, 42, 47, 47, 40, 46, 48, 44]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480393619Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"downstairs","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480404817Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"downstairs","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"[1]R flush_numbers[0..12]: [86, 86, 86, 81, 83, 74, 81, 81, 72, 78, 83, 75]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480415345Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"downstairs","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"[1]R generation[0..12]: [49, 49, 49, 47, 48, 42, 47, 47, 40, 46, 48, 44]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480425623Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"downstairs","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480436401Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"downstairs","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"[2]R flush_numbers[0..12]: [86, 86, 86, 81, 83, 74, 81, 81, 72, 78, 83, 75]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480446659Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"downstairs","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"[2]R generation[0..12]: [49, 49, 49, 47, 48, 42, 47, 47, 40, 46, 48, 44]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480456857Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"downstairs","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480467715Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"downstairs","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"Max found gen is 50","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480477034Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"downstairs","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"Generation requested: 50 >= found:50","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480486592Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"downstairs","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"Next flush: 87","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.48049592Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"downstairs","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480505418Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"":"downstairs","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480515296Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480524095Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"client":"0","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480535533Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"client":"1","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480545341Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"client":"2","":"downstairs","session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480555199Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480564377Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"f4f18a9f-12b3-44e1-ab7c-2b27467f3313 is now active with session: a4b5e210-912e-454d-8a2b-02ee09eb9286","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480574215Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480582794Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"session_id":"a4b5e210-912e-454d-8a2b-02ee09eb9286"} {"msg":"The guest has finished waiting for activation with:50","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480591802Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333} {"msg":"Activated sub_volume f4f18a9f-12b3-44e1-ab7c-2b27467f3313","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480599661Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480607719Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.480936278Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"task":"crutest"} 01/30 Write block 468 len 16384 data: 5 5 6 6 02/30 Read block 363 len 16384 03/30 Write block 509 len 36864 data: 3 3 3 3 3 3 5 6 6 04/30 Write block 260 len 16384 data: 5 4 3 3 05/30 Read block 31 len 24576 06/30 Read block 149 len 12288 07/30 Write block 537 len 8192 data: 3 3 08/30 Flush 09/30 Write block 191 len 12288 data: 6 6 7 10/30 Write block 590 len 20480 data: 8 7 8 9 9 11/30 Write block 514 len 8192 data: 4 6 12/30 Read block 440 len 28672 13/30 Read block 231 len 8192 14/30 Write block 195 len 36864 data: 6 6 5 3 3 2 2 3 3 15/30 Write block 174 len 20480 data: 5 5 4 4 6 16/30 Read block 291 len 12288 17/30 Flush 18/30 Write block 336 len 4096 data: 5 19/30 Read block 30 len 4096 20/30 Read block 448 len 8192 21/30 Read block 505 len 24576 22/30 Read block 2 len 40960 23/30 Read block 244 len 12288 24/30 Read block 303 len 36864 25/30 Read block 113 len 4096 26/30 Read block 13 len 24576 27/30 Write block 238 len 4096 data: 2 28/30 Read block 39 len 24576 29/30 Read block 44 len 16384 30/30 Read block 190 len 12288 ---------------------------------------------------------------- Crucible gen:50 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Sent Done false 1001 Acked Read 4 Done Sent Done false 1002 Acked Write 9 Done Sent Done false 1003 Acked Write 4 Done Sent Done false 1004 Acked Read 6 Done Sent Done false 1005 Acked Read 3 Done Sent Done false 1006 Acked Write 2 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Write 5 Done Sent Done false 1010 Acked Write 2 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Write 9 Done Sent Done false 1014 Acked Write 5 Done Sent Done false 1015 Acked Read 3 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Read 1 Done Sent Done false 1019 Acked Read 2 Done Sent Done false 1020 Acked Read 6 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Read 3 Done Sent Done false 1023 Acked Read 9 Done Sent Done false 1024 Acked Read 1 Done Sent Done false 1025 Acked Read 6 Done Sent Done false 1026 Acked Write 1 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Read 3 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 30 0 30 Done 30 0 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) None Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:04.489088441Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:04.489259179Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1333,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 49 49 49 86 86 86 F F F 1 020-039 49 49 49 86 86 86 F F F 2 040-059 49 49 49 86 86 86 F F F 3 060-079 47 47 47 81 81 81 F F F 4 080-099 48 48 48 83 83 83 F F F 5 100-119 42 42 42 74 74 74 F F F 6 120-139 47 47 47 81 81 81 F F F 7 140-159 47 47 47 81 81 81 F F F 8 160-179 50 40 50 88 72 88 F F F <--- 9 180-199 50 46 50 88 78 88 F F F <--- 10 200-219 50 48 50 88 83 88 F F F <--- 11 220-239 44 44 44 75 75 75 T F T <--- 12 240-259 42 42 42 74 74 74 F F F 13 260-279 50 46 50 87 78 87 F F F <--- 14 280-299 42 42 42 74 74 74 F F F 15 300-319 49 49 49 86 86 86 F F F 16 320-339 49 49 49 86 86 86 T F T <--- 17 340-359 35 35 35 64 64 64 F F F 18 360-379 46 46 46 79 79 79 F F F 19 380-399 48 48 48 84 84 84 F F F 20 400-419 41 41 41 73 73 73 F F F 21 420-439 49 49 49 86 86 86 F F F 22 440-459 46 46 46 80 80 80 F F F 23 460-479 50 47 50 87 81 87 F F F <--- 24 480-499 48 48 48 83 83 83 F F F 25 500-519 50 48 50 88 83 88 F F F <--- 26 520-539 50 45 50 87 77 87 F F F <--- 27 540-559 48 48 48 84 84 84 F F F 28 560-579 48 48 48 84 84 84 F F F 29 580-599 50 44 50 88 76 88 F F F <--- Max gen: 50, Max flush: 88 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 25 Verifying data now No change after verify Loop: 25 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 49 49 49 86 86 86 F F F 1 020-039 49 49 49 86 86 86 F F F 2 040-059 49 49 49 86 86 86 F F F 3 060-079 47 47 47 81 81 81 F F F 4 080-099 48 48 48 83 83 83 F F F 5 100-119 42 42 42 74 74 74 F F F 6 120-139 47 47 47 81 81 81 F F F 7 140-159 47 47 47 81 81 81 F F F 8 160-179 50 50 50 88 88 88 F F F 9 180-199 50 50 50 88 88 88 F F F 10 200-219 50 50 50 88 88 88 F F F 11 220-239 51 51 51 89 89 89 F F F 12 240-259 42 42 42 74 74 74 F F F 13 260-279 50 50 50 87 87 87 F F F 14 280-299 42 42 42 74 74 74 F F F 15 300-319 49 49 49 86 86 86 F F F 16 320-339 51 51 51 89 89 89 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 46 46 46 79 79 79 F F F 19 380-399 48 48 48 84 84 84 F F F 20 400-419 41 41 41 73 73 73 F F F 21 420-439 49 49 49 86 86 86 F F F 22 440-459 46 46 46 80 80 80 F F F 23 460-479 50 50 50 87 87 87 F F F 24 480-499 48 48 48 83 83 83 F F F 25 500-519 50 50 50 88 88 88 F F F 26 520-539 50 50 50 87 87 87 F F F 27 540-559 48 48 48 84 84 84 F F F 28 560-579 48 48 48 84 84 84 F F F 29 580-599 50 50 50 88 88 88 F F F Max gen: 51, Max flush: 89 Begin loop 26 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.73135952Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.73942055Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.739714695Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.73974121Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.739749899Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.739761286Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341} {"msg":"Crucible 6bdf7bd0-6870-4050-99ec-c195fea670e1 has session id: 5af76514-5575-4d71-89f5-81fefb903bf6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740140645Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"Upstairs opts: Upstairs UUID: 6bdf7bd0-6870-4050-99ec-c195fea670e1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740164671Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"Crucible stats registered with UUID: 6bdf7bd0-6870-4050-99ec-c195fea670e1","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740174359Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:08.740183347Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740341358Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"io task","client":"1","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740373612Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"io task","client":"0","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.74038449Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"io task","client":"2","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740510236Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"task":"crutest"} {"msg":"The guest has requested activation with gen:52","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740533722Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341} {"msg":"6bdf7bd0-6870-4050-99ec-c195fea670e1 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740572405Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740582183Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"client":"0","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740592131Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"client":"1","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740601759Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"client":"2","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740611147Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"io task","client":"2","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740621115Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"io task","client":"0","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740631004Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"io task","client":"1","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"ds_connection connected from Ok(127.0.0.1:33438)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740688693Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"io task","client":"2","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"ds_connection connected from Ok(127.0.0.1:57538)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740714068Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"io task","client":"1","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"ds_connection connected from Ok(127.0.0.1:50827)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.740725436Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"io task","client":"0","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741383693Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"client":"0","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741468837Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"client":"1","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741514908Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"client":"2","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741747674Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741766311Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"[0]R flush_numbers[0..12]: [86, 86, 86, 81, 83, 74, 81, 81, 88, 88, 88, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.74182326Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"downstairs","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"[0]R generation[0..12]: [49, 49, 49, 47, 48, 42, 47, 47, 50, 50, 50, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741834548Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"downstairs","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741845536Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"downstairs","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"[1]R flush_numbers[0..12]: [86, 86, 86, 81, 83, 74, 81, 81, 88, 88, 88, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741855864Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"downstairs","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"[1]R generation[0..12]: [49, 49, 49, 47, 48, 42, 47, 47, 50, 50, 50, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741866142Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"downstairs","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.74187704Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"downstairs","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"[2]R flush_numbers[0..12]: [86, 86, 86, 81, 83, 74, 81, 81, 88, 88, 88, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741889948Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"downstairs","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"[2]R generation[0..12]: [49, 49, 49, 47, 48, 42, 47, 47, 50, 50, 50, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741900366Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"downstairs","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741911074Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"downstairs","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"Max found gen is 52","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741920392Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"downstairs","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"Generation requested: 52 >= found:52","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.74192991Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"downstairs","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"Next flush: 90","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741939159Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"downstairs","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741948397Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"":"downstairs","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741957895Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741966713Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"client":"0","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741976402Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"client":"1","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.74198607Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"client":"2","":"downstairs","session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.741995768Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.742005086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"6bdf7bd0-6870-4050-99ec-c195fea670e1 is now active with session: 5af76514-5575-4d71-89f5-81fefb903bf6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.742015034Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.742023643Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"session_id":"5af76514-5575-4d71-89f5-81fefb903bf6"} {"msg":"The guest has finished waiting for activation with:52","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.742032641Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341} {"msg":"Activated sub_volume 6bdf7bd0-6870-4050-99ec-c195fea670e1","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.742041769Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.742049848Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.742366299Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"task":"crutest"} 01/30 Write block 401 len 32768 data: 3 4 4 5 4 4 4 3 02/30 Read block 98 len 40960 03/30 Write block 371 len 12288 data: 4 4 4 04/30 Read block 354 len 12288 05/30 Read block 376 len 20480 06/30 Flush 07/30 Write block 241 len 40960 data: 4 4 4 4 5 4 4 4 5 4 08/30 Write block 521 len 28672 data: 4 4 4 6 6 5 6 09/30 Write block 435 len 20480 data: 3 3 3 3 3 10/30 Read block 105 len 20480 11/30 Read block 192 len 12288 12/30 Write block 251 len 20480 data: 5 6 6 5 6 13/30 Write block 113 len 40960 data: 6 6 7 6 5 6 4 5 6 5 14/30 Read block 37 len 8192 15/30 Read block 314 len 40960 16/30 Read block 508 len 32768 17/30 Write block 65 len 24576 data: 5 5 5 5 4 3 18/30 Read block 555 len 32768 19/30 Read block 511 len 8192 20/30 Read block 286 len 4096 21/30 Read block 119 len 20480 22/30 Flush 23/30 Write block 326 len 32768 data: 5 5 5 5 5 5 6 5 24/30 Write block 380 len 4096 data: 8 25/30 Write block 456 len 36864 data: 4 3 3 3 3 3 3 3 5 26/30 Read block 447 len 16384 27/30 Read block 112 len 28672 28/30 Write block 277 len 4096 data: 6 29/30 Read block 484 len 36864 30/30 Write block 6 len 28672 data: 2 2 3 3 4 5 5 ---------------------------------------------------------------- Crucible gen:52 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Sent Done false 1001 Acked Read 10 Done Sent Done false 1002 Acked Write 3 Done Sent Done false 1003 Acked Read 3 Done Sent Done false 1004 Acked Read 5 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 10 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Read 5 Done Sent Done false 1010 Acked Read 3 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Read 2 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Read 8 Done Sent Done false 1016 Acked Write 6 Done Sent Done false 1017 Acked Read 8 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Read 5 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 8 Done Sent Done false 1023 Acked Write 1 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Read 4 Done Sent Done false 1026 Acked Read 7 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Read 9 Done Sent Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 1 32 Done 29 0 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) None Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:08.751369512Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:08.751532402Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1341,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 49 49 49 86 86 86 T F T <--- 1 020-039 49 49 49 86 86 86 F F F 2 040-059 49 49 49 86 86 86 F F F 3 060-079 52 47 52 91 81 91 F F F <--- 4 080-099 48 48 48 83 83 83 F F F 5 100-119 52 42 52 91 74 91 F F F <--- 6 120-139 52 47 52 91 81 91 F F F <--- 7 140-159 47 47 47 81 81 81 F F F 8 160-179 50 50 50 88 88 88 F F F 9 180-199 50 50 50 88 88 88 F F F 10 200-219 50 50 50 88 88 88 F F F 11 220-239 51 51 51 89 89 89 F F F 12 240-259 52 42 52 91 74 91 F F F <--- 13 260-279 50 50 50 87 87 87 T F T <--- 14 280-299 42 42 42 74 74 74 F F F 15 300-319 49 49 49 86 86 86 F F F 16 320-339 51 51 51 89 89 89 T F T <--- 17 340-359 35 35 35 64 64 64 F F F 18 360-379 52 46 52 90 79 90 F F F <--- 19 380-399 48 48 48 84 84 84 T F T <--- 20 400-419 52 41 52 90 73 90 F F F <--- 21 420-439 52 49 52 91 86 91 F F F <--- 22 440-459 46 46 46 80 80 80 T F T <--- 23 460-479 50 50 50 87 87 87 T F T <--- 24 480-499 48 48 48 83 83 83 F F F 25 500-519 50 50 50 88 88 88 F F F 26 520-539 52 50 52 91 87 91 F F F <--- 27 540-559 48 48 48 84 84 84 F F F 28 560-579 48 48 48 84 84 84 F F F 29 580-599 50 50 50 88 88 88 F F F Max gen: 52, Max flush: 91 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 26 Verifying data now No change after verify Loop: 26 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 53 53 53 92 92 92 F F F 1 020-039 49 49 49 86 86 86 F F F 2 040-059 49 49 49 86 86 86 F F F 3 060-079 52 52 52 91 91 91 F F F 4 080-099 48 48 48 83 83 83 F F F 5 100-119 52 52 52 91 91 91 F F F 6 120-139 52 52 52 91 91 91 F F F 7 140-159 47 47 47 81 81 81 F F F 8 160-179 50 50 50 88 88 88 F F F 9 180-199 50 50 50 88 88 88 F F F 10 200-219 50 50 50 88 88 88 F F F 11 220-239 51 51 51 89 89 89 F F F 12 240-259 52 52 52 91 91 91 F F F 13 260-279 53 53 53 92 92 92 F F F 14 280-299 42 42 42 74 74 74 F F F 15 300-319 49 49 49 86 86 86 F F F 16 320-339 53 53 53 92 92 92 F F F 17 340-359 35 35 35 64 64 64 F F F 18 360-379 52 52 52 90 90 90 F F F 19 380-399 53 53 53 92 92 92 F F F 20 400-419 52 52 52 90 90 90 F F F 21 420-439 52 52 52 91 91 91 F F F 22 440-459 53 53 53 92 92 92 F F F 23 460-479 53 53 53 92 92 92 F F F 24 480-499 48 48 48 83 83 83 F F F 25 500-519 50 50 50 88 88 88 F F F 26 520-539 52 52 52 91 91 91 F F F 27 540-559 48 48 48 84 84 84 F F F 28 560-579 48 48 48 84 84 84 F F F 29 580-599 50 50 50 88 88 88 F F F Max gen: 53, Max flush: 92 Begin loop 27 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.005632535Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.013816453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014109468Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014136213Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014144951Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014156399Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349} {"msg":"Crucible 010a9526-7ead-46cd-ac5f-466ec0f9d2db has session id: 9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014549655Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"Upstairs opts: Upstairs UUID: 010a9526-7ead-46cd-ac5f-466ec0f9d2db, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.01457873Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"Crucible stats registered with UUID: 010a9526-7ead-46cd-ac5f-466ec0f9d2db","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014590398Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:13.014599886Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014728092Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"io task","client":"1","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014761526Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"io task","client":"0","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014772484Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"io task","client":"2","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014915177Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"task":"crutest"} {"msg":"The guest has requested activation with gen:54","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014940212Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349} {"msg":"010a9526-7ead-46cd-ac5f-466ec0f9d2db active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014964148Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014973476Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"client":"0","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014983014Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"client":"1","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.014992412Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"client":"2","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.015001941Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"io task","client":"2","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.015011859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"io task","client":"0","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.015021817Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"io task","client":"1","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"ds_connection connected from Ok(127.0.0.1:32810)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.015084175Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"io task","client":"1","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"ds_connection connected from Ok(127.0.0.1:43358)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.01510967Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"io task","client":"0","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"ds_connection connected from Ok(127.0.0.1:39741)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.015182417Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"io task","client":"2","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.015767807Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"client":"0","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.015789923Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"client":"2","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.015803061Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"client":"1","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016106894Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016131919Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"[0]R flush_numbers[0..12]: [92, 86, 86, 91, 83, 91, 91, 81, 88, 88, 88, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016188528Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"downstairs","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"[0]R generation[0..12]: [53, 49, 49, 52, 48, 52, 52, 47, 50, 50, 50, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016200076Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"downstairs","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016211184Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"downstairs","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"[1]R flush_numbers[0..12]: [92, 86, 86, 91, 83, 91, 91, 81, 88, 88, 88, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016221722Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"downstairs","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"[1]R generation[0..12]: [53, 49, 49, 52, 48, 52, 52, 47, 50, 50, 50, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.01623201Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"downstairs","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016243208Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"downstairs","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"[2]R flush_numbers[0..12]: [92, 86, 86, 91, 83, 91, 91, 81, 88, 88, 88, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016253536Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"downstairs","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"[2]R generation[0..12]: [53, 49, 49, 52, 48, 52, 52, 47, 50, 50, 50, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016263914Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"downstairs","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016274752Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"downstairs","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"Max found gen is 54","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.01628431Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"downstairs","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"Generation requested: 54 >= found:54","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016293969Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"downstairs","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"Next flush: 93","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016305816Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"downstairs","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016320144Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"":"downstairs","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016329892Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.0163387Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"client":"0","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016348608Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"client":"1","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016358506Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"client":"2","":"downstairs","session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016368515Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016377903Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"010a9526-7ead-46cd-ac5f-466ec0f9d2db is now active with session: 9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016387781Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016396479Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"session_id":"9abd1f8e-9506-4d05-b1dc-dd74bd5ebe2f"} {"msg":"The guest has finished waiting for activation with:54","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016405588Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349} {"msg":"Activated sub_volume 010a9526-7ead-46cd-ac5f-466ec0f9d2db","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016413496Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016421495Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.016734536Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"task":"crutest"} 01/30 Write block 127 len 40960 data: 8 7 7 8 6 6 6 4 3 3 02/30 Read block 261 len 8192 03/30 Read block 525 len 40960 04/30 Flush 05/30 Write block 249 len 40960 data: 6 5 6 7 7 6 7 6 7 6 06/30 Read block 161 len 24576 07/30 Read block 157 len 4096 08/30 Read block 399 len 24576 09/30 Flush 10/30 Flush 11/30 Write block 562 len 16384 data: 8 8 6 7 12/30 Write block 364 len 24576 data: 4 4 3 2 2 2 13/30 Read block 78 len 8192 14/30 Read block 322 len 20480 15/30 Read block 347 len 40960 16/30 Write block 500 len 24576 data: 6 6 6 7 7 8 17/30 Write block 29 len 28672 data: 5 5 5 6 5 6 5 18/30 Write block 423 len 32768 data: 3 3 3 3 4 4 4 4 19/30 Write block 193 len 8192 data: 8 6 20/30 Read block 387 len 4096 21/30 Read block 324 len 16384 22/30 Flush 23/30 Write block 130 len 24576 data: 9 7 7 7 5 4 24/30 Read block 541 len 4096 25/30 Write block 212 len 28672 data: 7 6 7 7 7 7 8 26/30 Write block 299 len 16384 data: 3 3 3 5 27/30 Write block 349 len 24576 data: 4 4 4 3 3 3 28/30 Write block 246 len 12288 data: 5 5 5 29/30 Write block 10 len 40960 data: 5 6 6 4 4 5 6 5 7 7 30/30 Read block 16 len 16384 ---------------------------------------------------------------- Crucible gen:54 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Flush 0 Done Done Sent false 1004 Acked Write 10 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Read 1 Done Done Sent false 1007 Acked Read 6 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 4 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Read 2 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Read 10 Done Done Sent false 1015 Acked Write 6 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Write 8 Done Done Sent false 1018 Acked Write 2 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Read 4 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 6 Done Done Sent false 1023 Acked Read 1 Done Done Sent false 1024 Acked Write 7 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Write 6 Done Done Sent false 1027 Acked Write 3 Done Done Sent false 1028 Acked Write 10 Done Done Sent false 1029 Acked Read 4 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 27 27 Done 30 30 3 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1021)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:13.025501214Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:13.025680011Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1349,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 53 53 53 92 92 92 T T F <--- 1 020-039 54 54 49 96 96 86 F F F <--- 2 040-059 49 49 49 86 86 86 F F F 3 060-079 52 52 52 91 91 91 F F F 4 080-099 48 48 48 83 83 83 F F F 5 100-119 52 52 52 91 91 91 F F F 6 120-139 54 54 52 93 93 91 T T T <--- 7 140-159 47 47 47 81 81 81 F F F 8 160-179 50 50 50 88 88 88 F F F 9 180-199 54 54 50 96 96 88 F F F <--- 10 200-219 50 50 50 88 88 88 T T F <--- 11 220-239 51 51 51 89 89 89 F F F 12 240-259 54 54 52 94 94 91 T T F <--- 13 260-279 53 53 53 92 92 92 F F F 14 280-299 42 42 42 74 74 74 T T F <--- 15 300-319 49 49 49 86 86 86 T T F <--- 16 320-339 53 53 53 92 92 92 F F F 17 340-359 35 35 35 64 64 64 T T F <--- 18 360-379 54 54 52 96 96 90 F F F <--- 19 380-399 53 53 53 92 92 92 F F F 20 400-419 52 52 52 90 90 90 F F F 21 420-439 54 54 52 96 96 91 F F F <--- 22 440-459 53 53 53 92 92 92 F F F 23 460-479 53 53 53 92 92 92 F F F 24 480-499 48 48 48 83 83 83 F F F 25 500-519 54 54 50 96 96 88 F F F <--- 26 520-539 52 52 52 91 91 91 F F F 27 540-559 48 48 48 84 84 84 F F F 28 560-579 54 54 48 96 96 84 F F F <--- 29 580-599 50 50 50 88 88 88 F F F Max gen: 54, Max flush: 96 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 27 Verifying data now No change after verify Loop: 27 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 55 55 55 97 97 97 F F F 1 020-039 54 54 54 96 96 96 F F F 2 040-059 49 49 49 86 86 86 F F F 3 060-079 52 52 52 91 91 91 F F F 4 080-099 48 48 48 83 83 83 F F F 5 100-119 52 52 52 91 91 91 F F F 6 120-139 55 55 55 97 97 97 F F F 7 140-159 47 47 47 81 81 81 F F F 8 160-179 50 50 50 88 88 88 F F F 9 180-199 54 54 54 96 96 96 F F F 10 200-219 55 55 55 97 97 97 F F F 11 220-239 51 51 51 89 89 89 F F F 12 240-259 55 55 55 97 97 97 F F F 13 260-279 53 53 53 92 92 92 F F F 14 280-299 55 55 55 97 97 97 F F F 15 300-319 55 55 55 97 97 97 F F F 16 320-339 53 53 53 92 92 92 F F F 17 340-359 55 55 55 97 97 97 F F F 18 360-379 54 54 54 96 96 96 F F F 19 380-399 53 53 53 92 92 92 F F F 20 400-419 52 52 52 90 90 90 F F F 21 420-439 54 54 54 96 96 96 F F F 22 440-459 53 53 53 92 92 92 F F F 23 460-479 53 53 53 92 92 92 F F F 24 480-499 48 48 48 83 83 83 F F F 25 500-519 54 54 54 96 96 96 F F F 26 520-539 52 52 52 91 91 91 F F F 27 540-559 48 48 48 84 84 84 F F F 28 560-579 54 54 54 96 96 96 F F F 29 580-599 50 50 50 88 88 88 F F F Max gen: 55, Max flush: 97 Begin loop 28 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.276196935Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.284228381Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.284517387Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.284548141Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.28455653Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.284568028Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357} {"msg":"Crucible 7d29ce8a-9457-4e8c-8cf8-5a99a3f4be5d has session id: 4323722f-359d-414b-98a1-e4e3f5659874","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.284914833Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"Upstairs opts: Upstairs UUID: 7d29ce8a-9457-4e8c-8cf8-5a99a3f4be5d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.284942878Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"Crucible stats registered with UUID: 7d29ce8a-9457-4e8c-8cf8-5a99a3f4be5d","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.284953556Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:17.284962644Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285094809Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"io task","client":"1","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285111866Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"io task","client":"0","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285121834Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"io task","client":"2","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285276205Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"task":"crutest"} {"msg":"The guest has requested activation with gen:56","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285288573Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357} {"msg":"7d29ce8a-9457-4e8c-8cf8-5a99a3f4be5d active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.28530433Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285313368Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"client":"0","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285325216Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"client":"1","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285334844Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"client":"2","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285344292Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"io task","client":"2","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285354101Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"io task","client":"0","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285363839Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"io task","client":"1","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"ds_connection connected from Ok(127.0.0.1:44095)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285480057Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"io task","client":"1","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"ds_connection connected from Ok(127.0.0.1:57066)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285598165Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"io task","client":"0","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"ds_connection connected from Ok(127.0.0.1:46867)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.285611992Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"io task","client":"2","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286179846Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"client":"2","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286206061Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"client":"1","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286235995Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"client":"0","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286470141Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286495127Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"[0]R flush_numbers[0..12]: [97, 96, 86, 91, 83, 91, 97, 81, 88, 96, 97, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286541668Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"downstairs","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"[0]R generation[0..12]: [55, 54, 49, 52, 48, 52, 55, 47, 50, 54, 55, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286552596Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"downstairs","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286566214Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"downstairs","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"[1]R flush_numbers[0..12]: [97, 96, 86, 91, 83, 91, 97, 81, 88, 96, 97, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286576512Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"downstairs","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"[1]R generation[0..12]: [55, 54, 49, 52, 48, 52, 55, 47, 50, 54, 55, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.28658672Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"downstairs","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286597758Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"downstairs","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"[2]R flush_numbers[0..12]: [97, 96, 86, 91, 83, 91, 97, 81, 88, 96, 97, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286608166Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"downstairs","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"[2]R generation[0..12]: [55, 54, 49, 52, 48, 52, 55, 47, 50, 54, 55, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286618344Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"downstairs","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286629002Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"downstairs","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"Max found gen is 56","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.28663838Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"downstairs","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"Generation requested: 56 >= found:56","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286647958Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"downstairs","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"Next flush: 98","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286657186Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"downstairs","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286666405Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"":"downstairs","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286675953Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286684601Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"client":"0","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.28669411Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"client":"1","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286703518Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"client":"2","":"downstairs","session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286714626Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286723714Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"7d29ce8a-9457-4e8c-8cf8-5a99a3f4be5d is now active with session: 4323722f-359d-414b-98a1-e4e3f5659874","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286733632Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286742141Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"session_id":"4323722f-359d-414b-98a1-e4e3f5659874"} {"msg":"The guest has finished waiting for activation with:56","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286750979Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357} {"msg":"Activated sub_volume 7d29ce8a-9457-4e8c-8cf8-5a99a3f4be5d","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286759137Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.286767156Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.287077478Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"task":"crutest"} 01/30 Write block 337 len 40960 data: 4 5 4 4 4 4 5 5 4 3 02/30 Read block 348 len 12288 03/30 Write block 342 len 16384 data: 5 6 6 5 04/30 Read block 508 len 32768 05/30 Write block 176 len 32768 data: 5 5 7 5 5 4 6 7 06/30 Read block 324 len 36864 07/30 Read block 359 len 24576 08/30 Read block 341 len 24576 09/30 Write block 386 len 12288 data: 5 4 3 10/30 Write block 197 len 8192 data: 6 4 11/30 Read block 264 len 32768 12/30 Read block 421 len 28672 13/30 Read block 319 len 12288 14/30 Read block 147 len 32768 15/30 Read block 308 len 8192 16/30 Write block 125 len 40960 data: 9 9 9 8 8 10 8 8 8 6 17/30 Flush 18/30 Write block 356 len 4096 data: 4 19/30 Read block 481 len 32768 20/30 Read block 535 len 40960 21/30 Write block 178 len 24576 data: 8 6 6 5 7 8 22/30 Write block 299 len 28672 data: 4 4 4 6 7 9 7 23/30 Read block 444 len 4096 24/30 Read block 208 len 28672 25/30 Write block 119 len 16384 data: 5 6 7 6 26/30 Read block 203 len 12288 27/30 Write block 329 len 24576 data: 6 6 6 7 6 5 28/30 Write block 11 len 32768 data: 7 7 5 5 6 7 6 8 29/30 Read block 152 len 20480 30/30 Read block 101 len 28672 ---------------------------------------------------------------- Crucible gen:56 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Write 8 Done Done Done false 1005 Acked Read 9 Done Done Done false 1006 Acked Read 6 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Read 8 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Read 3 Done Sent Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Read 2 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Read 8 Done Sent Done false 1019 Acked Read 10 Done Sent Done false 1020 Acked Write 6 Done Sent Done false 1021 Acked Write 7 Done Sent Done false 1022 Acked Read 1 Done Sent Done false 1023 Acked Read 7 Done Sent Done false 1024 Acked Write 4 Done Sent Done false 1025 Acked Read 3 Done Sent Done false 1026 Acked Write 6 Done Sent Done false 1027 Acked Write 8 Done Sent Done false 1028 Acked Read 5 Done Sent Done false 1029 Acked Read 7 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 24 0 24 Done 30 6 30 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) None Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:17.296225125Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:17.296401901Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1357,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 55 55 55 97 97 97 T F T <--- 1 020-039 54 54 54 96 96 96 F F F 2 040-059 49 49 49 86 86 86 F F F 3 060-079 52 52 52 91 91 91 F F F 4 080-099 48 48 48 83 83 83 F F F 5 100-119 52 52 52 91 91 91 T F T <--- 6 120-139 56 55 56 98 97 98 T F T <--- 7 140-159 47 47 47 81 81 81 F F F 8 160-179 56 50 56 98 88 98 T T T <--- 9 180-199 56 54 56 98 96 98 T T T <--- 10 200-219 55 55 55 97 97 97 F F F 11 220-239 51 51 51 89 89 89 F F F 12 240-259 55 55 55 97 97 97 F F F 13 260-279 53 53 53 92 92 92 F F F 14 280-299 55 55 55 97 97 97 T F T <--- 15 300-319 55 55 55 97 97 97 T F T <--- 16 320-339 56 53 56 98 92 98 T T T <--- 17 340-359 56 55 56 98 97 98 T T T <--- 18 360-379 54 54 54 96 96 96 F F F 19 380-399 56 53 56 98 92 98 F F F <--- 20 400-419 52 52 52 90 90 90 F F F 21 420-439 54 54 54 96 96 96 F F F 22 440-459 53 53 53 92 92 92 F F F 23 460-479 53 53 53 92 92 92 F F F 24 480-499 48 48 48 83 83 83 F F F 25 500-519 54 54 54 96 96 96 F F F 26 520-539 52 52 52 91 91 91 F F F 27 540-559 48 48 48 84 84 84 F F F 28 560-579 54 54 54 96 96 96 F F F 29 580-599 50 50 50 88 88 88 F F F Max gen: 56, Max flush: 98 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 28 Verifying data now No change after verify Loop: 28 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 99 99 99 F F F 1 020-039 54 54 54 96 96 96 F F F 2 040-059 49 49 49 86 86 86 F F F 3 060-079 52 52 52 91 91 91 F F F 4 080-099 48 48 48 83 83 83 F F F 5 100-119 57 57 57 99 99 99 F F F 6 120-139 57 57 57 99 99 99 F F F 7 140-159 47 47 47 81 81 81 F F F 8 160-179 57 57 57 99 99 99 F F F 9 180-199 57 57 57 99 99 99 F F F 10 200-219 55 55 55 97 97 97 F F F 11 220-239 51 51 51 89 89 89 F F F 12 240-259 55 55 55 97 97 97 F F F 13 260-279 53 53 53 92 92 92 F F F 14 280-299 57 57 57 99 99 99 F F F 15 300-319 57 57 57 99 99 99 F F F 16 320-339 57 57 57 99 99 99 F F F 17 340-359 57 57 57 99 99 99 F F F 18 360-379 54 54 54 96 96 96 F F F 19 380-399 56 56 56 98 98 98 F F F 20 400-419 52 52 52 90 90 90 F F F 21 420-439 54 54 54 96 96 96 F F F 22 440-459 53 53 53 92 92 92 F F F 23 460-479 53 53 53 92 92 92 F F F 24 480-499 48 48 48 83 83 83 F F F 25 500-519 54 54 54 96 96 96 F F F 26 520-539 52 52 52 91 91 91 F F F 27 540-559 48 48 48 84 84 84 F F F 28 560-579 54 54 54 96 96 96 F F F 29 580-599 50 50 50 88 88 88 F F F Max gen: 57, Max flush: 99 Begin loop 29 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.538465379Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.546569782Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.546919046Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.54694974Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.546958419Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.546969967Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365} {"msg":"Crucible 4b24a28a-bfc0-48cb-8f66-7cabac688b8f has session id: 5034a0a9-53c6-430d-b5a0-c20f5992bda6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547294286Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"Upstairs opts: Upstairs UUID: 4b24a28a-bfc0-48cb-8f66-7cabac688b8f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547338638Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"Crucible stats registered with UUID: 4b24a28a-bfc0-48cb-8f66-7cabac688b8f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547349266Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:21.547358184Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547505626Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"io task","client":"0","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.54753695Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"io task","client":"1","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547548118Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"io task","client":"2","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547659618Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"task":"crutest"} {"msg":"The guest has requested activation with gen:58","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547676634Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365} {"msg":"4b24a28a-bfc0-48cb-8f66-7cabac688b8f active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547693521Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.54770236Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"client":"0","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547711778Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"client":"1","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547721336Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"client":"2","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547730804Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"io task","client":"2","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547740582Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"io task","client":"0","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.54775467Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"io task","client":"1","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"ds_connection connected from Ok(127.0.0.1:34424)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547802821Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"io task","client":"2","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"ds_connection connected from Ok(127.0.0.1:32993)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547815928Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"io task","client":"0","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"ds_connection connected from Ok(127.0.0.1:57367)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.547826336Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"io task","client":"1","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548469936Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"client":"0","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548489632Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"client":"2","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548571997Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"client":"1","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548842876Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548852444Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"[0]R flush_numbers[0..12]: [99, 96, 86, 91, 83, 99, 99, 81, 99, 99, 97, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548899255Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"downstairs","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"[0]R generation[0..12]: [57, 54, 49, 52, 48, 57, 57, 47, 57, 57, 55, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548910433Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"downstairs","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548921501Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"downstairs","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"[1]R flush_numbers[0..12]: [99, 96, 86, 91, 83, 99, 99, 81, 99, 99, 97, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548931879Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"downstairs","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"[1]R generation[0..12]: [57, 54, 49, 52, 48, 57, 57, 47, 57, 57, 55, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548942207Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"downstairs","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548953105Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"downstairs","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"[2]R flush_numbers[0..12]: [99, 96, 86, 91, 83, 99, 99, 81, 99, 99, 97, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548963493Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"downstairs","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"[2]R generation[0..12]: [57, 54, 49, 52, 48, 57, 57, 47, 57, 57, 55, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548975741Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"downstairs","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548986579Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"downstairs","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"Max found gen is 58","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.548996007Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"downstairs","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"Generation requested: 58 >= found:58","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549005655Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"downstairs","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"Next flush: 100","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549014964Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"downstairs","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549024302Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"":"downstairs","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.54903398Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549042748Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"client":"0","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549062465Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"client":"1","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549072113Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"client":"2","":"downstairs","session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549082021Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549091119Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"4b24a28a-bfc0-48cb-8f66-7cabac688b8f is now active with session: 5034a0a9-53c6-430d-b5a0-c20f5992bda6","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549100908Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549109686Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"session_id":"5034a0a9-53c6-430d-b5a0-c20f5992bda6"} {"msg":"The guest has finished waiting for activation with:58","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549118634Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365} {"msg":"Activated sub_volume 4b24a28a-bfc0-48cb-8f66-7cabac688b8f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549126933Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549136441Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.549437655Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"task":"crutest"} 01/30 Write block 286 len 20480 data: 4 4 4 5 5 02/30 Flush 03/30 Write block 29 len 20480 data: 6 6 6 7 6 04/30 Flush 05/30 Write block 475 len 4096 data: 3 06/30 Read block 284 len 32768 07/30 Flush 08/30 Write block 324 len 16384 data: 5 4 6 6 09/30 Write block 296 len 36864 data: 4 3 3 5 5 5 7 8 10 10/30 Read block 508 len 20480 11/30 Write block 387 len 32768 data: 5 4 2 3 4 5 6 5 12/30 Read block 395 len 20480 13/30 Write block 195 len 8192 data: 7 7 14/30 Read block 430 len 40960 15/30 Read block 208 len 32768 16/30 Read block 416 len 20480 17/30 Write block 31 len 12288 data: 7 8 7 18/30 Flush 19/30 Write block 73 len 8192 data: 3 3 20/30 Read block 334 len 24576 21/30 Read block 217 len 8192 22/30 Write block 261 len 28672 data: 5 4 4 2 2 3 6 23/30 Write block 52 len 24576 data: 3 3 5 5 7 6 24/30 Write block 61 len 28672 data: 6 8 5 4 6 6 6 25/30 Read block 467 len 20480 26/30 Write block 10 len 40960 data: 6 8 8 6 6 7 8 7 9 8 27/30 Read block 464 len 8192 28/30 Read block 111 len 24576 29/30 Write block 370 len 12288 data: 3 5 5 30/30 Read block 596 len 4096 ---------------------------------------------------------------- Crucible gen:58 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Sent Done false 1001 Acked Flush 0 Done Sent Done false 1002 Acked Write 5 Done Sent Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 1 Done Sent Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Flush 0 Done Sent Done false 1007 Acked Write 4 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Read 5 Done Sent Done false 1010 Acked Write 8 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Write 2 Done Sent Done false 1013 Acked Read 10 Done Sent Done false 1014 Acked Read 8 Done Sent Done false 1015 Acked Read 5 Done Sent Done false 1016 Acked Write 3 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 2 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Write 7 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Write 7 Done Sent Done false 1024 Acked Read 5 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Write 3 Done Sent Done false 1029 Acked Read 1 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 0 31 Done 29 0 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) None Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:21.558257163Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:21.55843399Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1365,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 57 57 57 99 99 99 T F T <--- 1 020-039 58 54 58 103 96 103 F F F <--- 2 040-059 49 49 49 86 86 86 T F T <--- 3 060-079 52 52 52 91 91 91 T F T <--- 4 080-099 48 48 48 83 83 83 F F F 5 100-119 57 57 57 99 99 99 F F F 6 120-139 57 57 57 99 99 99 F F F 7 140-159 47 47 47 81 81 81 F F F 8 160-179 57 57 57 99 99 99 F F F 9 180-199 58 57 58 103 99 103 F F F <--- 10 200-219 55 55 55 97 97 97 F F F 11 220-239 51 51 51 89 89 89 F F F 12 240-259 55 55 55 97 97 97 F F F 13 260-279 53 53 53 92 92 92 T F T <--- 14 280-299 58 57 58 103 99 103 F F F <--- 15 300-319 58 57 58 103 99 103 F F F <--- 16 320-339 58 57 58 103 99 103 F F F <--- 17 340-359 57 57 57 99 99 99 F F F 18 360-379 54 54 54 96 96 96 T F T <--- 19 380-399 58 56 58 103 98 103 F F F <--- 20 400-419 52 52 52 90 90 90 F F F 21 420-439 54 54 54 96 96 96 F F F 22 440-459 53 53 53 92 92 92 F F F 23 460-479 58 53 58 102 92 102 F F F <--- 24 480-499 48 48 48 83 83 83 F F F 25 500-519 54 54 54 96 96 96 F F F 26 520-539 52 52 52 91 91 91 F F F 27 540-559 48 48 48 84 84 84 F F F 28 560-579 54 54 54 96 96 96 F F F 29 580-599 50 50 50 88 88 88 F F F Max gen: 58, Max flush: 103 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 29 Verifying data now No change after verify Loop: 29 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 59 59 59 104 104 104 F F F 1 020-039 58 58 58 103 103 103 F F F 2 040-059 59 59 59 104 104 104 F F F 3 060-079 59 59 59 104 104 104 F F F 4 080-099 48 48 48 83 83 83 F F F 5 100-119 57 57 57 99 99 99 F F F 6 120-139 57 57 57 99 99 99 F F F 7 140-159 47 47 47 81 81 81 F F F 8 160-179 57 57 57 99 99 99 F F F 9 180-199 58 58 58 103 103 103 F F F 10 200-219 55 55 55 97 97 97 F F F 11 220-239 51 51 51 89 89 89 F F F 12 240-259 55 55 55 97 97 97 F F F 13 260-279 59 59 59 104 104 104 F F F 14 280-299 58 58 58 103 103 103 F F F 15 300-319 58 58 58 103 103 103 F F F 16 320-339 58 58 58 103 103 103 F F F 17 340-359 57 57 57 99 99 99 F F F 18 360-379 59 59 59 104 104 104 F F F 19 380-399 58 58 58 103 103 103 F F F 20 400-419 52 52 52 90 90 90 F F F 21 420-439 54 54 54 96 96 96 F F F 22 440-459 53 53 53 92 92 92 F F F 23 460-479 58 58 58 102 102 102 F F F 24 480-499 48 48 48 83 83 83 F F F 25 500-519 54 54 54 96 96 96 F F F 26 520-539 52 52 52 91 91 91 F F F 27 540-559 48 48 48 84 84 84 F F F 28 560-579 54 54 54 96 96 96 F F F 29 580-599 50 50 50 88 88 88 F F F Max gen: 59, Max flush: 104 Begin loop 30 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.811221429Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.819414524Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.819697002Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.819729126Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.819737554Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.819749142Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373} {"msg":"Crucible 608912a9-353c-4391-8089-40d1957ae585 has session id: a2be6a1b-a392-4e4a-a4c4-6fd002c6487f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820105255Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"Upstairs opts: Upstairs UUID: 608912a9-353c-4391-8089-40d1957ae585, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.82013495Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"Crucible stats registered with UUID: 608912a9-353c-4391-8089-40d1957ae585","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820145937Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:25.820155126Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.82029139Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"io task","client":"0","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820308597Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"io task","client":"1","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820320725Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"io task","client":"2","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.82045037Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"task":"crutest"} {"msg":"The guest has requested activation with gen:60","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820467007Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373} {"msg":"608912a9-353c-4391-8089-40d1957ae585 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820500261Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820509839Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"client":"0","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820519368Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"client":"1","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820528886Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"client":"2","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820538184Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"io task","client":"2","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820548072Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"io task","client":"0","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820558Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"io task","client":"1","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"ds_connection connected from Ok(127.0.0.1:58517)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820598293Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"io task","client":"0","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"ds_connection connected from Ok(127.0.0.1:57983)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820609971Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"io task","client":"1","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"ds_connection connected from Ok(127.0.0.1:64276)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.820623878Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"io task","client":"2","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821283714Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"client":"0","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821300421Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"client":"1","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821384016Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"client":"2","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821611803Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821621221Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"[0]R flush_numbers[0..12]: [104, 103, 104, 104, 83, 99, 99, 81, 99, 103, 97, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821669532Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"downstairs","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"[0]R generation[0..12]: [59, 58, 59, 59, 48, 57, 57, 47, 57, 58, 55, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.82168047Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"downstairs","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821691438Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"downstairs","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"[1]R flush_numbers[0..12]: [104, 103, 104, 104, 83, 99, 99, 81, 99, 103, 97, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821701836Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"downstairs","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"[1]R generation[0..12]: [59, 58, 59, 59, 48, 57, 57, 47, 57, 58, 55, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821712044Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"downstairs","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821722922Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"downstairs","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"[2]R flush_numbers[0..12]: [104, 103, 104, 104, 83, 99, 99, 81, 99, 103, 97, 89]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.82173322Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"downstairs","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"[2]R generation[0..12]: [59, 58, 59, 59, 48, 57, 57, 47, 57, 58, 55, 51]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821743468Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"downstairs","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821754246Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"downstairs","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"Max found gen is 60","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821763505Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"downstairs","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"Generation requested: 60 >= found:60","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821773103Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"downstairs","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"Next flush: 105","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821782431Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"downstairs","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821793069Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"":"downstairs","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821803017Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821811905Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"client":"0","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821821704Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"client":"1","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821831512Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"client":"2","":"downstairs","session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.82184144Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821850558Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"608912a9-353c-4391-8089-40d1957ae585 is now active with session: a2be6a1b-a392-4e4a-a4c4-6fd002c6487f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821865445Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821874164Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"session_id":"a2be6a1b-a392-4e4a-a4c4-6fd002c6487f"} {"msg":"The guest has finished waiting for activation with:60","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821883132Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373} {"msg":"Activated sub_volume 608912a9-353c-4391-8089-40d1957ae585","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821890921Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.821899029Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.822224808Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"task":"crutest"} 01/30 Write block 212 len 36864 data: 8 7 8 8 8 8 9 6 6 02/30 Write block 342 len 24576 data: 6 7 7 6 4 3 03/30 Read block 353 len 32768 04/30 Write block 375 len 36864 data: 7 7 7 7 7 9 8 8 8 05/30 Read block 198 len 32768 06/30 Write block 397 len 28672 data: 7 6 5 3 4 5 5 07/30 Read block 308 len 20480 08/30 Write block 97 len 4096 data: 3 09/30 Write block 202 len 12288 data: 4 4 4 10/30 Write block 152 len 24576 data: 3 3 3 3 3 3 11/30 Read block 189 len 16384 12/30 Write block 174 len 4096 data: 6 13/30 Read block 465 len 40960 14/30 Write block 558 len 28672 data: 8 10 9 8 9 9 7 15/30 Read block 510 len 4096 16/30 Write block 166 len 36864 data: 5 4 4 3 3 3 5 5 7 17/30 Write block 388 len 20480 data: 5 3 4 5 6 18/30 Read block 529 len 40960 19/30 Read block 440 len 24576 20/30 Write block 84 len 8192 data: 6 7 21/30 Flush 22/30 Write block 349 len 36864 data: 5 5 5 4 4 4 3 5 4 23/30 Write block 12 len 20480 data: 9 7 7 8 9 24/30 Read block 454 len 24576 25/30 Read block 182 len 16384 26/30 Flush 27/30 Write block 278 len 20480 data: 5 6 5 3 2 28/30 Write block 431 len 20480 data: 2 2 2 2 4 29/30 Write block 178 len 8192 data: 9 7 30/30 Read block 557 len 8192 ---------------------------------------------------------------- Crucible gen:60 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Write 9 Done Done Done false 1004 Acked Read 8 Done Done Done false 1005 Acked Write 7 Done Done Done false 1006 Acked Read 5 Done Done Done false 1007 Acked Write 1 Done Done Done false 1008 Acked Write 3 Done Done Done false 1009 Acked Write 6 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Write 1 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Write 7 Done Done Sent false 1014 Acked Read 1 Done Done Sent false 1015 Acked Write 9 Done Done Sent false 1016 Acked Write 5 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Read 6 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Write 5 Done Done Sent false 1023 Acked Read 6 Done Done Sent false 1024 Acked Read 4 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 5 Done Done Sent false 1027 Acked Write 5 Done Sent Sent false 1028 Acked Write 2 Done Sent Sent false 1029 Acked Read 2 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 3 21 24 Done 30 27 9 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:25.831129231Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:25.831274993Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1373,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 59 106 106 104 F F F <--- 1 020-039 58 58 58 103 103 103 F F F 2 040-059 59 59 59 104 104 104 F F F 3 060-079 59 59 59 104 104 104 F F F 4 080-099 60 60 48 105 105 83 F F T <--- 5 100-119 57 57 57 99 99 99 F F F 6 120-139 57 57 57 99 99 99 F F F 7 140-159 60 60 47 105 105 81 F F F <--- 8 160-179 60 60 57 105 105 99 T T F <--- 9 180-199 58 58 58 103 103 103 F F F 10 200-219 60 60 55 105 105 97 F F T <--- 11 220-239 60 60 51 105 105 89 F F T <--- 12 240-259 55 55 55 97 97 97 F F F 13 260-279 59 59 59 104 104 104 T T F <--- 14 280-299 58 58 58 103 103 103 T T F <--- 15 300-319 58 58 58 103 103 103 F F F 16 320-339 58 58 58 103 103 103 F F F 17 340-359 60 60 57 106 106 99 F F T <--- 18 360-379 60 60 59 105 105 104 F F T <--- 19 380-399 60 60 58 105 105 103 F F T <--- 20 400-419 60 60 52 105 105 90 F F T <--- 21 420-439 54 54 54 96 96 96 T T F <--- 22 440-459 53 53 53 92 92 92 F F F 23 460-479 58 58 58 102 102 102 F F F 24 480-499 48 48 48 83 83 83 F F F 25 500-519 54 54 54 96 96 96 F F F 26 520-539 52 52 52 91 91 91 F F F 27 540-559 60 60 48 105 105 84 F F F <--- 28 560-579 60 60 54 105 105 96 F F F <--- 29 580-599 50 50 50 88 88 88 F F F Max gen: 60, Max flush: 106 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 30 Verifying data now No change after verify Loop: 30 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 106 106 106 F F F 1 020-039 58 58 58 103 103 103 F F F 2 040-059 59 59 59 104 104 104 F F F 3 060-079 59 59 59 104 104 104 F F F 4 080-099 60 60 60 105 105 105 F F F 5 100-119 57 57 57 99 99 99 F F F 6 120-139 57 57 57 99 99 99 F F F 7 140-159 60 60 60 105 105 105 F F F 8 160-179 61 61 61 107 107 107 F F F 9 180-199 58 58 58 103 103 103 F F F 10 200-219 60 60 60 105 105 105 F F F 11 220-239 60 60 60 105 105 105 F F F 12 240-259 55 55 55 97 97 97 F F F 13 260-279 61 61 61 107 107 107 F F F 14 280-299 61 61 61 107 107 107 F F F 15 300-319 58 58 58 103 103 103 F F F 16 320-339 58 58 58 103 103 103 F F F 17 340-359 60 60 60 106 106 106 F F F 18 360-379 60 60 60 105 105 105 F F F 19 380-399 60 60 60 105 105 105 F F F 20 400-419 60 60 60 105 105 105 F F F 21 420-439 61 61 61 107 107 107 F F F 22 440-459 53 53 53 92 92 92 F F F 23 460-479 58 58 58 102 102 102 F F F 24 480-499 48 48 48 83 83 83 F F F 25 500-519 54 54 54 96 96 96 F F F 26 520-539 52 52 52 91 91 91 F F F 27 540-559 60 60 60 105 105 105 F F F 28 560-579 60 60 60 105 105 105 F F F 29 580-599 50 50 50 88 88 88 F F F Max gen: 61, Max flush: 107 Begin loop 31 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.087258734Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:30.094400787Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.094455566Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.101348255Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.101633632Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.101661857Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.101670305Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.101681663Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381} {"msg":"Crucible cb3c1334-1a9f-436c-80af-61475e512349 has session id: 91a1d78c-2a93-410d-9203-a07b5d7ac29f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102074249Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"Upstairs opts: Upstairs UUID: cb3c1334-1a9f-436c-80af-61475e512349, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102103544Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"Crucible stats registered with UUID: cb3c1334-1a9f-436c-80af-61475e512349","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102115032Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:30.1021243Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102253376Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"io task","client":"1","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.10228353Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"io task","client":"0","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102294538Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"io task","client":"2","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102431063Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"task":"crutest"} {"msg":"The guest has requested activation with gen:62","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102455718Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381} {"msg":"cb3c1334-1a9f-436c-80af-61475e512349 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102479804Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102491211Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"client":"0","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.10250084Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"client":"1","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102510338Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"client":"2","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102519776Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"io task","client":"2","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102529524Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"io task","client":"0","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102555419Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"io task","client":"1","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"ds_connection connected from Ok(127.0.0.1:50087)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102566117Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"io task","client":"2","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"ds_connection connected from Ok(127.0.0.1:58516)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102615998Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"io task","client":"1","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"ds_connection connected from Ok(127.0.0.1:46859)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.102641443Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"io task","client":"0","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103317947Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"client":"2","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103356429Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"client":"1","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.10340379Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"client":"0","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103628238Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103670091Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"[0]R flush_numbers[0..12]: [106, 103, 104, 104, 105, 99, 99, 105, 107, 103, 105, 105]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103720671Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"downstairs","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"[0]R generation[0..12]: [60, 58, 59, 59, 60, 57, 57, 60, 61, 58, 60, 60]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103733719Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"downstairs","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103744797Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"downstairs","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"[1]R flush_numbers[0..12]: [106, 103, 104, 104, 105, 99, 99, 105, 107, 103, 105, 105]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103755285Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"downstairs","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"[1]R generation[0..12]: [60, 58, 59, 59, 60, 57, 57, 60, 61, 58, 60, 60]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103765513Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"downstairs","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103776201Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"downstairs","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"[2]R flush_numbers[0..12]: [106, 103, 104, 104, 105, 99, 99, 105, 107, 103, 105, 105]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103786689Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"downstairs","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"[2]R generation[0..12]: [60, 58, 59, 59, 60, 57, 57, 60, 61, 58, 60, 60]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103796987Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"downstairs","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103807815Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"downstairs","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"Max found gen is 62","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103817043Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"downstairs","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"Generation requested: 62 >= found:62","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103826651Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"downstairs","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"Next flush: 108","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103836109Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"downstairs","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103845288Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"":"downstairs","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103854906Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103863464Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"client":"0","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103874502Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"client":"1","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103883961Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"client":"2","":"downstairs","session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103893639Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103902807Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"cb3c1334-1a9f-436c-80af-61475e512349 is now active with session: 91a1d78c-2a93-410d-9203-a07b5d7ac29f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103912825Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103921653Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"session_id":"91a1d78c-2a93-410d-9203-a07b5d7ac29f"} {"msg":"The guest has finished waiting for activation with:62","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103930722Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381} {"msg":"Activated sub_volume cb3c1334-1a9f-436c-80af-61475e512349","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.10393889Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.103947079Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.104238344Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"task":"crutest"} 01/30 Write block 453 len 20480 data: 2 3 2 5 4 02/30 Write block 410 len 12288 data: 3 2 2 03/30 Write block 281 len 4096 data: 4 04/30 Write block 123 len 28672 data: 8 10 10 10 10 9 9 05/30 Read block 566 len 24576 06/30 Read block 344 len 4096 07/30 Write block 30 len 12288 data: 7 8 9 08/30 Write block 188 len 36864 data: 3 5 5 7 7 9 7 8 8 09/30 Read block 354 len 32768 10/30 Read block 413 len 40960 11/30 Read block 217 len 36864 12/30 Flush 13/30 Write block 135 len 36864 data: 5 4 5 7 7 7 7 7 6 14/30 Read block 544 len 8192 15/30 Write block 288 len 12288 data: 5 6 6 16/30 Write block 351 len 28672 data: 6 5 5 5 4 6 5 17/30 Read block 492 len 12288 18/30 Read block 333 len 8192 19/30 Write block 346 len 4096 data: 5 20/30 Write block 55 len 28672 data: 6 8 7 6 7 7 7 21/30 Read block 576 len 36864 22/30 Write block 383 len 16384 data: 9 7 5 6 23/30 Read block 232 len 4096 24/30 Read block 213 len 12288 25/30 Read block 509 len 36864 26/30 Write block 502 len 12288 data: 7 8 8 27/30 Write block 107 len 24576 data: 5 5 6 6 6 6 28/30 Write block 315 len 28672 data: 7 7 8 5 6 7 7 29/30 Read block 460 len 4096 30/30 Read block 10 len 4096 ---------------------------------------------------------------- Crucible gen:62 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Sent Done Done false 1001 Acked Write 3 Sent Done Done false 1002 Acked Write 1 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Read 1 Sent Done Done false 1006 Acked Write 3 Sent Done Done false 1007 Acked Write 9 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Read 10 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Write 3 Sent Done Done false 1015 Acked Write 7 Sent Done Done false 1016 Acked Read 3 Sent Done Done false 1017 Acked Read 2 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Write 7 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Read 1 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Read 9 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Write 6 Sent Done Done false 1027 Acked Write 7 Sent Done Done false 1028 Acked Read 1 Sent Done Done false 1029 Acked Read 1 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1011)) Some(JobId(1011)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:30.112147713Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:30.112328489Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1381,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 106 106 106 F F F 1 020-039 58 62 62 103 108 108 F F F <--- 2 040-059 59 59 59 104 104 104 F T T <--- 3 060-079 59 59 59 104 104 104 F T T <--- 4 080-099 60 60 60 105 105 105 F F F 5 100-119 57 57 57 99 99 99 F T T <--- 6 120-139 57 62 62 99 108 108 F T T <--- 7 140-159 60 60 60 105 105 105 F T T <--- 8 160-179 61 61 61 107 107 107 F F F 9 180-199 58 62 62 103 108 108 F F F <--- 10 200-219 60 60 60 105 105 105 F F F 11 220-239 60 60 60 105 105 105 F F F 12 240-259 55 55 55 97 97 97 F F F 13 260-279 61 61 61 107 107 107 F F F 14 280-299 61 62 62 107 108 108 F T T <--- 15 300-319 58 58 58 103 103 103 F T T <--- 16 320-339 58 58 58 103 103 103 F T T <--- 17 340-359 60 60 60 106 106 106 F T T <--- 18 360-379 60 60 60 105 105 105 F F F 19 380-399 60 60 60 105 105 105 F T T <--- 20 400-419 60 62 62 105 108 108 F F F <--- 21 420-439 61 61 61 107 107 107 F F F 22 440-459 53 62 62 92 108 108 F F F <--- 23 460-479 58 58 58 102 102 102 F F F 24 480-499 48 48 48 83 83 83 F F F 25 500-519 54 54 54 96 96 96 F T T <--- 26 520-539 52 52 52 91 91 91 F F F 27 540-559 60 60 60 105 105 105 F F F 28 560-579 60 60 60 105 105 105 F F F 29 580-599 50 50 50 88 88 88 F F F Max gen: 62, Max flush: 108 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 31 Verifying data now No change after verify Loop: 31 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 60 60 106 106 106 F F F 1 020-039 62 62 62 108 108 108 F F F 2 040-059 63 63 63 109 109 109 F F F 3 060-079 63 63 63 109 109 109 F F F 4 080-099 60 60 60 105 105 105 F F F 5 100-119 63 63 63 109 109 109 F F F 6 120-139 63 63 63 109 109 109 F F F 7 140-159 63 63 63 109 109 109 F F F 8 160-179 61 61 61 107 107 107 F F F 9 180-199 62 62 62 108 108 108 F F F 10 200-219 60 60 60 105 105 105 F F F 11 220-239 60 60 60 105 105 105 F F F 12 240-259 55 55 55 97 97 97 F F F 13 260-279 61 61 61 107 107 107 F F F 14 280-299 63 63 63 109 109 109 F F F 15 300-319 63 63 63 109 109 109 F F F 16 320-339 63 63 63 109 109 109 F F F 17 340-359 63 63 63 109 109 109 F F F 18 360-379 60 60 60 105 105 105 F F F 19 380-399 63 63 63 109 109 109 F F F 20 400-419 62 62 62 108 108 108 F F F 21 420-439 61 61 61 107 107 107 F F F 22 440-459 62 62 62 108 108 108 F F F 23 460-479 58 58 58 102 102 102 F F F 24 480-499 48 48 48 83 83 83 F F F 25 500-519 63 63 63 109 109 109 F F F 26 520-539 52 52 52 91 91 91 F F F 27 540-559 60 60 60 105 105 105 F F F 28 560-579 60 60 60 105 105 105 F F F 29 580-599 50 50 50 88 88 88 F F F Max gen: 63, Max flush: 109 Begin loop 32 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.375274916Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:34.382512321Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.382571729Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.389523747Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.389812793Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.389841348Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.389852456Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.389864104Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389} {"msg":"Crucible 18fab471-c6bb-4ef9-aca6-d333162c75ad has session id: d1764cec-33c4-43b9-9c4a-e7d5dad1bb16","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390221287Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"Upstairs opts: Upstairs UUID: 18fab471-c6bb-4ef9-aca6-d333162c75ad, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390252781Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"Crucible stats registered with UUID: 18fab471-c6bb-4ef9-aca6-d333162c75ad","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390263759Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:34.390273017Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.39041761Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"io task","client":"2","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390449804Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"io task","client":"0","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390461112Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"io task","client":"1","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390590268Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"task":"crutest"} {"msg":"The guest has requested activation with gen:64","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390617403Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389} {"msg":"18fab471-c6bb-4ef9-aca6-d333162c75ad active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.39063371Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390642648Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"client":"0","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390652536Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"client":"1","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390662324Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"client":"2","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390674222Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"io task","client":"2","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.39068455Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"io task","client":"0","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390717794Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"io task","client":"1","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"ds_connection connected from Ok(127.0.0.1:51394)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390752737Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"io task","client":"0","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"ds_connection connected from Ok(127.0.0.1:65407)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390783032Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"io task","client":"2","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"ds_connection connected from Ok(127.0.0.1:57580)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.390823014Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"io task","client":"1","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391471653Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"client":"1","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.3914885Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"client":"0","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391561696Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"client":"2","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391760279Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391779625Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"[0]R flush_numbers[0..12]: [106, 108, 109, 109, 105, 109, 109, 109, 107, 108, 105, 105]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391838574Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"downstairs","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"[0]R generation[0..12]: [60, 62, 63, 63, 60, 63, 63, 63, 61, 62, 60, 60]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391849722Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"downstairs","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.39186065Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"downstairs","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"[1]R flush_numbers[0..12]: [106, 108, 109, 109, 105, 109, 109, 109, 107, 108, 105, 105]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391873927Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"downstairs","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"[1]R generation[0..12]: [60, 62, 63, 63, 60, 63, 63, 63, 61, 62, 60, 60]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391884105Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"downstairs","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391894883Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"downstairs","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"[2]R flush_numbers[0..12]: [106, 108, 109, 109, 105, 109, 109, 109, 107, 108, 105, 105]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391905371Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"downstairs","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"[2]R generation[0..12]: [60, 62, 63, 63, 60, 63, 63, 63, 61, 62, 60, 60]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.39191568Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"downstairs","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391926467Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"downstairs","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"Max found gen is 64","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391935846Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"downstairs","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"Generation requested: 64 >= found:64","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391945554Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"downstairs","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"Next flush: 110","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391954882Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"downstairs","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.39196443Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"":"downstairs","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391973979Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391982587Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"client":"0","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.391992315Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"client":"1","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.392002313Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"client":"2","":"downstairs","session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.392012401Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.392023139Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"18fab471-c6bb-4ef9-aca6-d333162c75ad is now active with session: d1764cec-33c4-43b9-9c4a-e7d5dad1bb16","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.392032938Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.392041696Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"session_id":"d1764cec-33c4-43b9-9c4a-e7d5dad1bb16"} {"msg":"The guest has finished waiting for activation with:64","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.392050714Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389} {"msg":"Activated sub_volume 18fab471-c6bb-4ef9-aca6-d333162c75ad","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.392058633Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.392066631Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.392396759Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"task":"crutest"} 01/30 Write block 589 len 12288 data: 8 9 8 02/30 Read block 375 len 12288 03/30 Read block 196 len 8192 04/30 Write block 100 len 24576 data: 4 3 3 4 4 4 05/30 Write block 334 len 12288 data: 6 6 6 06/30 Flush 07/30 Flush 08/30 Write block 232 len 28672 data: 4 4 4 3 4 3 3 09/30 Write block 585 len 24576 data: 3 5 7 8 9 10 10/30 Write block 298 len 32768 data: 4 6 6 6 8 9 11 8 11/30 Flush 12/30 Flush 13/30 Write block 437 len 12288 data: 4 4 4 14/30 Flush 15/30 Write block 6 len 24576 data: 3 3 4 4 7 9 16/30 Write block 195 len 8192 data: 9 9 17/30 Read block 461 len 32768 18/30 Write block 103 len 20480 data: 5 5 5 3 6 19/30 Read block 452 len 36864 20/30 Write block 38 len 40960 data: 6 6 6 7 9 7 6 6 6 4 21/30 Write block 106 len 8192 data: 4 7 22/30 Flush 23/30 Write block 488 len 24576 data: 5 8 8 8 7 6 24/30 Write block 303 len 36864 data: 10 12 9 9 11 10 10 11 10 25/30 Read block 197 len 4096 26/30 Write block 554 len 16384 data: 5 5 5 8 27/30 Read block 442 len 16384 28/30 Write block 110 len 12288 data: 7 7 7 29/30 Write block 130 len 28672 data: 11 9 9 9 7 6 5 30/30 Read block 389 len 4096 ---------------------------------------------------------------- Crucible gen:64 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 3 Sent Done Done false 1002 Acked Read 2 Sent Done Done false 1003 Acked Write 6 Sent Done Done false 1004 Acked Write 3 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 7 Sent Done Done false 1008 Acked Write 6 Sent Done Done false 1009 Acked Write 8 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 6 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Read 8 Sent Done Done false 1017 Acked Write 5 Sent Done Done false 1018 Acked Read 9 Sent Done Done false 1019 Acked Write 10 Sent Done Done false 1020 Acked Write 2 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 6 Sent Done Done false 1023 Acked Write 9 Sent Done Done false 1024 Acked Read 1 Sent Done Done false 1025 Acked Write 4 Sent Done Done false 1026 Acked Read 4 Sent Done Done false 1027 Acked Write 3 Sent Done Done false 1028 Acked Write 7 Sent Done Done false 1029 Acked Read 1 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 0 29 Done 1 30 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:34.401231015Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:34.401428068Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1389,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 60 64 64 106 115 115 F F F <--- 1 020-039 62 64 64 108 115 115 F F F <--- 2 040-059 63 64 64 109 115 115 F F F <--- 3 060-079 63 63 63 109 109 109 F F F 4 080-099 60 60 60 105 105 105 F F F 5 100-119 63 64 64 109 115 115 F T T <--- 6 120-139 63 63 63 109 109 109 F T T <--- 7 140-159 63 63 63 109 109 109 F F F 8 160-179 61 61 61 107 107 107 F F F 9 180-199 62 64 64 108 115 115 F F F <--- 10 200-219 60 60 60 105 105 105 F F F 11 220-239 60 64 64 105 112 112 F F F <--- 12 240-259 55 55 55 97 97 97 F F F 13 260-279 61 61 61 107 107 107 F F F 14 280-299 63 64 64 109 112 112 F F F <--- 15 300-319 63 64 64 109 112 112 F T T <--- 16 320-339 63 64 64 109 110 110 F F F <--- 17 340-359 63 63 63 109 109 109 F F F 18 360-379 60 60 60 105 105 105 F F F 19 380-399 63 63 63 109 109 109 F F F 20 400-419 62 62 62 108 108 108 F F F 21 420-439 61 64 64 107 114 114 F F F <--- 22 440-459 62 62 62 108 108 108 F F F 23 460-479 58 58 58 102 102 102 F F F 24 480-499 48 48 48 83 83 83 F T T <--- 25 500-519 63 63 63 109 109 109 F F F 26 520-539 52 52 52 91 91 91 F F F 27 540-559 60 60 60 105 105 105 F T T <--- 28 560-579 60 60 60 105 105 105 F F F 29 580-599 50 64 64 88 112 112 T F F <--- Max gen: 64, Max flush: 115 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 32 Verifying data now No change after verify Loop: 32 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 64 115 115 115 F F F 1 020-039 64 64 64 115 115 115 F F F 2 040-059 64 64 64 115 115 115 F F F 3 060-079 63 63 63 109 109 109 F F F 4 080-099 60 60 60 105 105 105 F F F 5 100-119 65 65 65 116 116 116 F F F 6 120-139 65 65 65 116 116 116 F F F 7 140-159 63 63 63 109 109 109 F F F 8 160-179 61 61 61 107 107 107 F F F 9 180-199 64 64 64 115 115 115 F F F 10 200-219 60 60 60 105 105 105 F F F 11 220-239 64 64 64 112 112 112 F F F 12 240-259 55 55 55 97 97 97 F F F 13 260-279 61 61 61 107 107 107 F F F 14 280-299 64 64 64 112 112 112 F F F 15 300-319 65 65 65 116 116 116 F F F 16 320-339 64 64 64 110 110 110 F F F 17 340-359 63 63 63 109 109 109 F F F 18 360-379 60 60 60 105 105 105 F F F 19 380-399 63 63 63 109 109 109 F F F 20 400-419 62 62 62 108 108 108 F F F 21 420-439 64 64 64 114 114 114 F F F 22 440-459 62 62 62 108 108 108 F F F 23 460-479 58 58 58 102 102 102 F F F 24 480-499 65 65 65 116 116 116 F F F 25 500-519 63 63 63 109 109 109 F F F 26 520-539 52 52 52 91 91 91 F F F 27 540-559 65 65 65 116 116 116 F F F 28 560-579 60 60 60 105 105 105 F F F 29 580-599 64 64 64 112 112 112 F F F Max gen: 65, Max flush: 116 Begin loop 33 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.658713715Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.666983666Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.66728598Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.667313934Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.667322453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.667334021Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397} {"msg":"Crucible f3182e9f-96b1-41d6-8a6d-59683ab6b5d3 has session id: cdcaab42-32d6-482a-b5b9-69bec1e3c6b4","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.667679236Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"Upstairs opts: Upstairs UUID: f3182e9f-96b1-41d6-8a6d-59683ab6b5d3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.6677092Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"Crucible stats registered with UUID: f3182e9f-96b1-41d6-8a6d-59683ab6b5d3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.667722788Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:38.667732046Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.667881688Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"io task","client":"1","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.667899525Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"io task","client":"0","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.667909443Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"io task","client":"2","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668074022Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"task":"crutest"} {"msg":"The guest has requested activation with gen:66","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668098188Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397} {"msg":"f3182e9f-96b1-41d6-8a6d-59683ab6b5d3 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668107416Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668116574Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"client":"0","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668126552Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"client":"1","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.66813616Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"client":"2","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668157556Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"io task","client":"2","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668168104Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"io task","client":"0","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668178003Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"io task","client":"1","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:34555)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668213666Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"io task","client":"1","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:56930)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668226704Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"io task","client":"0","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"ds_connection connected from Ok(127.0.0.1:41258)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668236992Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"io task","client":"2","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668925163Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"client":"0","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668945569Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"client":"1","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.668956667Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"client":"2","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.66925903Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669269028Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"[0]R flush_numbers[0..12]: [115, 115, 115, 109, 105, 116, 116, 109, 107, 115, 105, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669303852Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"downstairs","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"[0]R generation[0..12]: [64, 64, 64, 63, 60, 65, 65, 63, 61, 64, 60, 64]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.66931487Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"downstairs","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669325678Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"downstairs","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"[1]R flush_numbers[0..12]: [115, 115, 115, 109, 105, 116, 116, 109, 107, 115, 105, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669336216Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"downstairs","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"[1]R generation[0..12]: [64, 64, 64, 63, 60, 65, 65, 63, 61, 64, 60, 64]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669346434Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"downstairs","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669357312Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"downstairs","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"[2]R flush_numbers[0..12]: [115, 115, 115, 109, 105, 116, 116, 109, 107, 115, 105, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.66936795Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"downstairs","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"[2]R generation[0..12]: [64, 64, 64, 63, 60, 65, 65, 63, 61, 64, 60, 64]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669380707Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"downstairs","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669391565Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"downstairs","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"Max found gen is 66","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669400984Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"downstairs","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"Generation requested: 66 >= found:66","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669410672Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"downstairs","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"Next flush: 117","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.66942016Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"downstairs","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669429548Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"":"downstairs","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669439276Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669447985Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"client":"0","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669457613Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"client":"1","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669467281Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"client":"2","":"downstairs","session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669477179Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669486398Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"f3182e9f-96b1-41d6-8a6d-59683ab6b5d3 is now active with session: cdcaab42-32d6-482a-b5b9-69bec1e3c6b4","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669496476Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669505294Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"session_id":"cdcaab42-32d6-482a-b5b9-69bec1e3c6b4"} {"msg":"The guest has finished waiting for activation with:66","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669514532Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397} {"msg":"Activated sub_volume f3182e9f-96b1-41d6-8a6d-59683ab6b5d3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669522591Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669530669Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.669863177Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"task":"crutest"} 01/30 Write block 250 len 28672 data: 6 7 8 8 7 8 7 02/30 Read block 561 len 8192 03/30 Read block 226 len 40960 04/30 Read block 254 len 16384 05/30 Read block 135 len 36864 06/30 Write block 542 len 28672 data: 4 6 6 6 5 4 4 07/30 Read block 291 len 4096 08/30 Write block 439 len 16384 data: 5 5 5 5 09/30 Write block 209 len 28672 data: 8 8 6 9 8 9 9 10/30 Flush 11/30 Write block 182 len 4096 data: 8 12/30 Write block 329 len 16384 data: 7 7 7 8 13/30 Read block 302 len 12288 14/30 Read block 179 len 8192 15/30 Read block 346 len 36864 16/30 Write block 241 len 36864 data: 5 5 5 5 6 6 6 6 7 17/30 Read block 564 len 32768 18/30 Write block 210 len 32768 data: 9 7 10 9 10 10 9 9 19/30 Read block 125 len 40960 20/30 Write block 422 len 4096 data: 3 21/30 Write block 524 len 24576 data: 7 7 6 7 6 6 22/30 Write block 207 len 16384 data: 6 7 9 10 23/30 Write block 274 len 12288 data: 5 6 6 24/30 Read block 375 len 40960 25/30 Write block 364 len 12288 data: 5 5 4 26/30 Read block 462 len 12288 27/30 Read block 58 len 36864 28/30 Write block 482 len 36864 data: 6 6 6 6 7 7 6 9 9 29/30 Read block 127 len 32768 30/30 Write block 440 len 16384 data: 6 6 6 6 ---------------------------------------------------------------- Crucible gen:66 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 2 Done Done Done false 1002 Acked Read 10 Done Done Sent false 1003 Acked Read 4 Done Done Sent false 1004 Acked Read 9 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Read 1 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Write 7 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Write 4 Done Done Sent false 1012 Acked Read 3 Done Done Sent false 1013 Acked Read 2 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Write 9 Done Done Sent false 1016 Acked Read 8 Done Done Sent false 1017 Acked Write 8 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Write 1 Done Done Sent false 1020 Acked Write 6 Done Done Sent false 1021 Acked Write 4 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Read 10 Done Done Sent false 1024 Acked Write 3 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Read 9 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 8 Done Done Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 28 30 Done 29 29 2 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1009)) Some(JobId(1009)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:38.678912032Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:38.679105956Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1397,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 64 115 115 115 F F F 1 020-039 64 64 64 115 115 115 F F F 2 040-059 64 64 64 115 115 115 F F F 3 060-079 63 63 63 109 109 109 F F F 4 080-099 60 60 60 105 105 105 F F F 5 100-119 65 65 65 116 116 116 F F F 6 120-139 65 65 65 116 116 116 F F F 7 140-159 63 63 63 109 109 109 F F F 8 160-179 61 61 61 107 107 107 F F F 9 180-199 64 64 64 115 115 115 T T F <--- 10 200-219 66 66 60 117 117 105 T T F <--- 11 220-239 64 64 64 112 112 112 F F F 12 240-259 66 66 55 117 117 97 T T T <--- 13 260-279 61 61 61 107 107 107 T T F <--- 14 280-299 64 64 64 112 112 112 F F F 15 300-319 65 65 65 116 116 116 F F F 16 320-339 64 64 64 110 110 110 T T F <--- 17 340-359 63 63 63 109 109 109 F F F 18 360-379 60 60 60 105 105 105 T T F <--- 19 380-399 63 63 63 109 109 109 F F F 20 400-419 62 62 62 108 108 108 F F F 21 420-439 66 66 64 117 117 114 T T F <--- 22 440-459 66 66 62 117 117 108 T T F <--- 23 460-479 58 58 58 102 102 102 F F F 24 480-499 65 65 65 116 116 116 T T F <--- 25 500-519 63 63 63 109 109 109 F F F 26 520-539 52 52 52 91 91 91 T T F <--- 27 540-559 66 66 65 117 117 116 F F F <--- 28 560-579 60 60 60 105 105 105 F F F 29 580-599 64 64 64 112 112 112 F F F Max gen: 66, Max flush: 117 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 33 Verifying data now No change after verify Loop: 33 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 64 115 115 115 F F F 1 020-039 64 64 64 115 115 115 F F F 2 040-059 64 64 64 115 115 115 F F F 3 060-079 63 63 63 109 109 109 F F F 4 080-099 60 60 60 105 105 105 F F F 5 100-119 65 65 65 116 116 116 F F F 6 120-139 65 65 65 116 116 116 F F F 7 140-159 63 63 63 109 109 109 F F F 8 160-179 61 61 61 107 107 107 F F F 9 180-199 67 67 67 118 118 118 F F F 10 200-219 67 67 67 118 118 118 F F F 11 220-239 64 64 64 112 112 112 F F F 12 240-259 67 67 67 118 118 118 F F F 13 260-279 67 67 67 118 118 118 F F F 14 280-299 64 64 64 112 112 112 F F F 15 300-319 65 65 65 116 116 116 F F F 16 320-339 67 67 67 118 118 118 F F F 17 340-359 63 63 63 109 109 109 F F F 18 360-379 67 67 67 118 118 118 F F F 19 380-399 63 63 63 109 109 109 F F F 20 400-419 62 62 62 108 108 108 F F F 21 420-439 67 67 67 118 118 118 F F F 22 440-459 67 67 67 118 118 118 F F F 23 460-479 58 58 58 102 102 102 F F F 24 480-499 67 67 67 118 118 118 F F F 25 500-519 63 63 63 109 109 109 F F F 26 520-539 67 67 67 118 118 118 F F F 27 540-559 66 66 66 117 117 117 F F F 28 560-579 60 60 60 105 105 105 F F F 29 580-599 64 64 64 112 112 112 F F F Max gen: 67, Max flush: 118 Begin loop 34 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.928860503Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:42.936099298Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.936163905Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.943141609Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.943428845Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.943458129Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.943466428Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.943477836Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405} {"msg":"Crucible e0bafe4e-13a0-4ded-a248-0f67a12324b2 has session id: 33c5bf60-df42-45f5-8c85-9b03348fe5f5","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.943825621Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"Upstairs opts: Upstairs UUID: e0bafe4e-13a0-4ded-a248-0f67a12324b2, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.943854085Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"Crucible stats registered with UUID: e0bafe4e-13a0-4ded-a248-0f67a12324b2","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.943864843Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:42.943874031Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944019914Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"io task","client":"1","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944037211Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"io task","client":"0","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944047349Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"io task","client":"2","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944216617Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"task":"crutest"} {"msg":"The guest has requested activation with gen:68","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944240353Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405} {"msg":"e0bafe4e-13a0-4ded-a248-0f67a12324b2 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.9442573Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944266338Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"client":"0","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944276006Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"client":"1","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944285474Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"client":"2","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944295063Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"io task","client":"2","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944305191Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"io task","client":"0","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944315249Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"io task","client":"1","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:65294)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944381686Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"io task","client":"2","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:37594)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944457702Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"io task","client":"1","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"ds_connection connected from Ok(127.0.0.1:65449)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.944532888Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"io task","client":"0","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945116249Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"client":"1","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945134205Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"client":"2","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945145153Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"client":"0","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945412543Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945443358Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"[0]R flush_numbers[0..12]: [115, 115, 115, 109, 105, 116, 116, 109, 107, 118, 118, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945532841Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"downstairs","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"[0]R generation[0..12]: [64, 64, 64, 63, 60, 65, 65, 63, 61, 67, 67, 64]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945543869Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"downstairs","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945554867Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"downstairs","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"[1]R flush_numbers[0..12]: [115, 115, 115, 109, 105, 116, 116, 109, 107, 118, 118, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945565375Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"downstairs","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"[1]R generation[0..12]: [64, 64, 64, 63, 60, 65, 65, 63, 61, 67, 67, 64]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945575693Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"downstairs","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945586411Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"downstairs","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"[2]R flush_numbers[0..12]: [115, 115, 115, 109, 105, 116, 116, 109, 107, 118, 118, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945596829Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"downstairs","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"[2]R generation[0..12]: [64, 64, 64, 63, 60, 65, 65, 63, 61, 67, 67, 64]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945606927Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"downstairs","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945617755Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"downstairs","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"Max found gen is 68","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945627103Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"downstairs","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"Generation requested: 68 >= found:68","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945638151Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"downstairs","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"Next flush: 119","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945647399Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"downstairs","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945656718Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"":"downstairs","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945666326Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945675154Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"client":"0","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945684692Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"client":"1","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945694181Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"client":"2","":"downstairs","session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945703999Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945713047Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"e0bafe4e-13a0-4ded-a248-0f67a12324b2 is now active with session: 33c5bf60-df42-45f5-8c85-9b03348fe5f5","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945722965Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945731624Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"session_id":"33c5bf60-df42-45f5-8c85-9b03348fe5f5"} {"msg":"The guest has finished waiting for activation with:68","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945740522Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405} {"msg":"Activated sub_volume e0bafe4e-13a0-4ded-a248-0f67a12324b2","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.94574875Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.945757159Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.946096825Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"task":"crutest"} 01/30 Write block 498 len 40960 data: 6 6 7 7 8 9 9 9 7 6 02/30 Write block 384 len 12288 data: 8 6 7 03/30 Read block 146 len 16384 04/30 Write block 262 len 12288 data: 5 5 3 05/30 Read block 255 len 8192 06/30 Read block 310 len 24576 07/30 Read block 511 len 16384 08/30 Write block 192 len 8192 data: 8 10 09/30 Read block 210 len 32768 10/30 Read block 61 len 12288 11/30 Write block 126 len 4096 data: 11 12/30 Read block 508 len 16384 13/30 Write block 526 len 16384 data: 7 8 7 7 14/30 Flush 15/30 Flush 16/30 Write block 250 len 20480 data: 7 8 9 9 8 17/30 Read block 561 len 40960 18/30 Write block 371 len 32768 data: 6 6 5 6 8 8 8 8 19/30 Read block 592 len 28672 20/30 Flush 21/30 Write block 45 len 4096 data: 7 22/30 Read block 113 len 8192 23/30 Write block 154 len 24576 data: 4 4 4 4 4 4 24/30 Write block 442 len 16384 data: 7 7 5 5 25/30 Read block 285 len 32768 26/30 Read block 48 len 16384 27/30 Write block 247 len 36864 data: 7 7 8 8 9 10 10 9 9 28/30 Read block 144 len 8192 29/30 Write block 100 len 28672 data: 5 4 4 6 6 6 5 30/30 Read block 353 len 28672 ---------------------------------------------------------------- Crucible gen:68 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 3 Sent Done Done false 1002 Acked Read 4 Sent Done Done false 1003 Acked Write 3 Sent Done Done false 1004 Acked Read 2 Sent Done Done false 1005 Acked Read 6 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Read 8 Sent Done Done false 1009 Acked Read 3 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Read 4 Sent Done Done false 1012 Acked Write 4 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Read 10 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Read 7 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 1 Sent Done Done false 1021 Acked Read 2 Sent Done Done false 1022 Acked Write 6 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Read 4 Sent Done Done false 1026 Acked Write 9 Sent Done Done false 1027 Acked Read 2 Sent Done Done false 1028 Acked Write 7 Sent Sent Done false 1029 Acked Read 7 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 2 0 31 Done 1 28 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:42.955047719Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:42.955229015Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1405,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 64 115 115 115 F F F 1 020-039 64 64 64 115 115 115 F F F 2 040-059 64 64 64 115 115 115 F T T <--- 3 060-079 63 63 63 109 109 109 F F F 4 080-099 60 60 60 105 105 105 F F F 5 100-119 65 65 65 116 116 116 F T T <--- 6 120-139 65 68 68 116 119 119 F F F <--- 7 140-159 63 63 63 109 109 109 F T T <--- 8 160-179 61 61 61 107 107 107 F F F 9 180-199 67 68 68 118 119 119 F F F <--- 10 200-219 67 67 67 118 118 118 F F F 11 220-239 64 64 64 112 112 112 F F F 12 240-259 67 68 68 118 121 121 F T T <--- 13 260-279 67 68 68 118 119 119 F F F <--- 14 280-299 64 64 64 112 112 112 F F F 15 300-319 65 65 65 116 116 116 F F F 16 320-339 67 67 67 118 118 118 F F F 17 340-359 63 63 63 109 109 109 F F F 18 360-379 67 68 68 118 121 121 F F F <--- 19 380-399 63 68 68 109 119 119 F F F <--- 20 400-419 62 62 62 108 108 108 F F F 21 420-439 67 67 67 118 118 118 F F F 22 440-459 67 67 67 118 118 118 F T T <--- 23 460-479 58 58 58 102 102 102 F F F 24 480-499 67 68 68 118 119 119 T F F <--- 25 500-519 63 68 68 109 119 119 T F F <--- 26 520-539 67 68 68 118 119 119 F F F <--- 27 540-559 66 66 66 117 117 117 F F F 28 560-579 60 60 60 105 105 105 F F F 29 580-599 64 64 64 112 112 112 F F F Max gen: 68, Max flush: 121 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 34 Verifying data now No change after verify Loop: 34 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 64 115 115 115 F F F 1 020-039 64 64 64 115 115 115 F F F 2 040-059 69 69 69 122 122 122 F F F 3 060-079 63 63 63 109 109 109 F F F 4 080-099 60 60 60 105 105 105 F F F 5 100-119 69 69 69 122 122 122 F F F 6 120-139 68 68 68 119 119 119 F F F 7 140-159 69 69 69 122 122 122 F F F 8 160-179 61 61 61 107 107 107 F F F 9 180-199 68 68 68 119 119 119 F F F 10 200-219 67 67 67 118 118 118 F F F 11 220-239 64 64 64 112 112 112 F F F 12 240-259 69 69 69 122 122 122 F F F 13 260-279 68 68 68 119 119 119 F F F 14 280-299 64 64 64 112 112 112 F F F 15 300-319 65 65 65 116 116 116 F F F 16 320-339 67 67 67 118 118 118 F F F 17 340-359 63 63 63 109 109 109 F F F 18 360-379 68 68 68 121 121 121 F F F 19 380-399 68 68 68 119 119 119 F F F 20 400-419 62 62 62 108 108 108 F F F 21 420-439 67 67 67 118 118 118 F F F 22 440-459 69 69 69 122 122 122 F F F 23 460-479 58 58 58 102 102 102 F F F 24 480-499 68 68 68 119 119 119 F F F 25 500-519 68 68 68 119 119 119 F F F 26 520-539 68 68 68 119 119 119 F F F 27 540-559 66 66 66 117 117 117 F F F 28 560-579 60 60 60 105 105 105 F F F 29 580-599 64 64 64 112 112 112 F F F Max gen: 69, Max flush: 122 Begin loop 35 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.210167682Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.218522577Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.218809443Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.218835278Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.218843787Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.218855285Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413} {"msg":"Crucible 695b226e-066c-4dfa-b9f7-85f630fcd33f has session id: c507843d-40e4-448b-bcd5-66776860f13e","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219250111Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"Upstairs opts: Upstairs UUID: 695b226e-066c-4dfa-b9f7-85f630fcd33f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219271747Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"Crucible stats registered with UUID: 695b226e-066c-4dfa-b9f7-85f630fcd33f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219281445Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:47.219290503Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219451843Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"io task","client":"1","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219484387Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"io task","client":"0","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219495395Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"io task","client":"2","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219606174Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"task":"crutest"} {"msg":"The guest has requested activation with gen:70","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219621831Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413} {"msg":"695b226e-066c-4dfa-b9f7-85f630fcd33f active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219656704Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219668282Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"client":"0","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.21967798Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"client":"1","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219687859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"client":"2","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219697437Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"io task","client":"2","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219707395Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"io task","client":"0","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219717083Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"io task","client":"1","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"ds_connection connected from Ok(127.0.0.1:56311)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219763185Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"io task","client":"0","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"ds_connection connected from Ok(127.0.0.1:55451)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219774342Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"io task","client":"1","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"ds_connection connected from Ok(127.0.0.1:63652)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.219792579Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"io task","client":"2","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220440068Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"client":"0","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220457374Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"client":"2","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.22058587Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"client":"1","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220760938Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220770536Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"[0]R flush_numbers[0..12]: [115, 115, 122, 109, 105, 122, 119, 122, 107, 119, 118, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220903131Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"downstairs","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"[0]R generation[0..12]: [64, 64, 69, 63, 60, 69, 68, 69, 61, 68, 67, 64]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220916718Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"downstairs","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220927616Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"downstairs","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"[1]R flush_numbers[0..12]: [115, 115, 122, 109, 105, 122, 119, 122, 107, 119, 118, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220938034Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"downstairs","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"[1]R generation[0..12]: [64, 64, 69, 63, 60, 69, 68, 69, 61, 68, 67, 64]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220948283Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"downstairs","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220958971Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"downstairs","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"[2]R flush_numbers[0..12]: [115, 115, 122, 109, 105, 122, 119, 122, 107, 119, 118, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220969359Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"downstairs","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"[2]R generation[0..12]: [64, 64, 69, 63, 60, 69, 68, 69, 61, 68, 67, 64]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220979487Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"downstairs","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220990215Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"downstairs","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"Max found gen is 70","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.220999453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"downstairs","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"Generation requested: 70 >= found:70","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.221009061Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"downstairs","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"Next flush: 123","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.221018429Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"downstairs","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.221027548Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"":"downstairs","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.221037086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.221045864Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"client":"0","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.221056862Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"client":"1","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.22106637Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"client":"2","":"downstairs","session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.221075949Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.221084857Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"695b226e-066c-4dfa-b9f7-85f630fcd33f is now active with session: c507843d-40e4-448b-bcd5-66776860f13e","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.221094565Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.221103314Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"session_id":"c507843d-40e4-448b-bcd5-66776860f13e"} {"msg":"The guest has finished waiting for activation with:70","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.221112322Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413} {"msg":"Activated sub_volume 695b226e-066c-4dfa-b9f7-85f630fcd33f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.22112026Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.221128359Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.221445809Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"task":"crutest"} 01/30 Write block 387 len 32768 data: 6 6 4 5 6 7 7 6 02/30 Write block 176 len 16384 data: 6 6 10 8 03/30 Read block 238 len 32768 04/30 Flush 05/30 Write block 557 len 4096 data: 9 06/30 Flush 07/30 Write block 48 len 16384 data: 5 5 4 4 08/30 Flush 09/30 Write block 334 len 36864 data: 7 7 7 5 6 5 5 5 7 10/30 Write block 318 len 24576 data: 6 7 8 8 5 4 11/30 Read block 54 len 40960 12/30 Write block 185 len 40960 data: 6 7 4 4 6 6 8 9 11 8 13/30 Read block 564 len 28672 14/30 Read block 30 len 4096 15/30 Read block 217 len 28672 16/30 Write block 323 len 16384 data: 5 6 5 7 17/30 Read block 264 len 20480 18/30 Write block 71 len 4096 data: 2 19/30 Flush 20/30 Write block 151 len 28672 data: 3 4 4 5 5 5 5 21/30 Read block 552 len 12288 22/30 Read block 357 len 8192 23/30 Read block 168 len 16384 24/30 Read block 588 len 28672 25/30 Write block 449 len 36864 data: 2 2 3 2 3 4 3 6 5 26/30 Read block 292 len 20480 27/30 Read block 307 len 16384 28/30 Write block 29 len 16384 data: 7 8 9 10 29/30 Read block 577 len 16384 30/30 Write block 325 len 4096 data: 6 ---------------------------------------------------------------- Crucible gen:70 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Read 8 Done Done Done false 1003 Acked Flush 0 Done Sent Done false 1004 Acked Write 1 Done Sent Done false 1005 Acked Flush 0 Done Sent Done false 1006 Acked Write 4 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 9 Done Sent Done false 1009 Acked Write 6 Done Sent Done false 1010 Acked Read 10 Done Sent Done false 1011 Acked Write 10 Done Sent Done false 1012 Acked Read 7 Done Sent Done false 1013 Acked Read 1 Done Sent Done false 1014 Acked Read 7 Done Sent Done false 1015 Acked Write 4 Done Sent Done false 1016 Acked Read 5 Done Sent Done false 1017 Acked Write 1 Done Sent Done false 1018 Acked Flush 0 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Read 3 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Read 4 Done Sent Done false 1023 Acked Read 7 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Read 5 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 27 1 29 Done 29 3 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) None Some(JobId(1018)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:47.230332065Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:47.2305187Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1413,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 64 115 115 115 F F F 1 020-039 64 64 64 115 115 115 T F T <--- 2 040-059 70 69 70 125 122 125 F F F <--- 3 060-079 70 63 70 126 109 126 F F F <--- 4 080-099 60 60 60 105 105 105 F F F 5 100-119 69 69 69 122 122 122 F F F 6 120-139 68 68 68 119 119 119 F F F 7 140-159 69 69 69 122 122 122 T F T <--- 8 160-179 70 61 70 123 107 123 F T F <--- 9 180-199 70 68 70 126 119 126 F F F <--- 10 200-219 67 67 67 118 118 118 F F F 11 220-239 64 64 64 112 112 112 F F F 12 240-259 69 69 69 122 122 122 F F F 13 260-279 68 68 68 119 119 119 F F F 14 280-299 64 64 64 112 112 112 F F F 15 300-319 70 65 70 126 116 126 F F F <--- 16 320-339 70 67 70 126 118 126 T F T <--- 17 340-359 70 63 70 126 109 126 F F F <--- 18 360-379 68 68 68 121 121 121 F F F 19 380-399 70 68 70 123 119 123 F T F <--- 20 400-419 62 62 62 108 108 108 F F F 21 420-439 67 67 67 118 118 118 F F F 22 440-459 69 69 69 122 122 122 T F T <--- 23 460-479 58 58 58 102 102 102 F F F 24 480-499 68 68 68 119 119 119 F F F 25 500-519 68 68 68 119 119 119 F F F 26 520-539 68 68 68 119 119 119 F F F 27 540-559 70 66 70 124 117 124 F F F <--- 28 560-579 60 60 60 105 105 105 F F F 29 580-599 64 64 64 112 112 112 F F F Max gen: 70, Max flush: 126 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 35 Verifying data now No change after verify Loop: 35 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 64 64 64 115 115 115 F F F 1 020-039 71 71 71 127 127 127 F F F 2 040-059 70 70 70 125 125 125 F F F 3 060-079 70 70 70 126 126 126 F F F 4 080-099 60 60 60 105 105 105 F F F 5 100-119 69 69 69 122 122 122 F F F 6 120-139 68 68 68 119 119 119 F F F 7 140-159 71 71 71 127 127 127 F F F 8 160-179 70 70 70 123 123 123 F F F 9 180-199 70 70 70 126 126 126 F F F 10 200-219 67 67 67 118 118 118 F F F 11 220-239 64 64 64 112 112 112 F F F 12 240-259 69 69 69 122 122 122 F F F 13 260-279 68 68 68 119 119 119 F F F 14 280-299 64 64 64 112 112 112 F F F 15 300-319 70 70 70 126 126 126 F F F 16 320-339 71 71 71 127 127 127 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 68 68 68 121 121 121 F F F 19 380-399 70 70 70 123 123 123 F F F 20 400-419 62 62 62 108 108 108 F F F 21 420-439 67 67 67 118 118 118 F F F 22 440-459 71 71 71 127 127 127 F F F 23 460-479 58 58 58 102 102 102 F F F 24 480-499 68 68 68 119 119 119 F F F 25 500-519 68 68 68 119 119 119 F F F 26 520-539 68 68 68 119 119 119 F F F 27 540-559 70 70 70 124 124 124 F F F 28 560-579 60 60 60 105 105 105 F F F 29 580-599 64 64 64 112 112 112 F F F Max gen: 71, Max flush: 127 Begin loop 36 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.481217831Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.489415086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.489703492Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.489730017Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.489738815Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.489750153Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421} {"msg":"Crucible bc9f7600-804f-47ab-9fe6-ed33c6bef6da has session id: c2bc73ff-42e1-4496-820a-8a48b92a2df1","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490102727Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"Upstairs opts: Upstairs UUID: bc9f7600-804f-47ab-9fe6-ed33c6bef6da, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490152078Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"Crucible stats registered with UUID: bc9f7600-804f-47ab-9fe6-ed33c6bef6da","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490162576Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:51.490171564Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490310928Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"io task","client":"1","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490341162Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"io task","client":"0","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.49035204Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"io task","client":"2","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490490014Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"task":"crutest"} {"msg":"The guest has requested activation with gen:72","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490525248Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421} {"msg":"bc9f7600-804f-47ab-9fe6-ed33c6bef6da active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490534006Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490542834Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"client":"0","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490552633Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"client":"1","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490562231Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"client":"2","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490571779Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"io task","client":"2","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490581627Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"io task","client":"0","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490593615Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"io task","client":"1","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"ds_connection connected from Ok(127.0.0.1:55132)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490655753Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"io task","client":"2","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"ds_connection connected from Ok(127.0.0.1:63523)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490682058Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"io task","client":"1","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"ds_connection connected from Ok(127.0.0.1:64986)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.490693536Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"io task","client":"0","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491331497Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"client":"1","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491357322Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"client":"0","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491492956Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"client":"2","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491623042Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491641369Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"[0]R flush_numbers[0..12]: [115, 127, 125, 126, 105, 122, 119, 127, 123, 126, 118, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491758037Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"downstairs","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"[0]R generation[0..12]: [64, 71, 70, 70, 60, 69, 68, 71, 70, 70, 67, 64]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491769185Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"downstairs","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491780113Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"downstairs","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"[1]R flush_numbers[0..12]: [115, 127, 125, 126, 105, 122, 119, 127, 123, 126, 118, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491790701Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"downstairs","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"[1]R generation[0..12]: [64, 71, 70, 70, 60, 69, 68, 71, 70, 70, 67, 64]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491800899Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"downstairs","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491814176Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"downstairs","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"[2]R flush_numbers[0..12]: [115, 127, 125, 126, 105, 122, 119, 127, 123, 126, 118, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491824684Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"downstairs","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"[2]R generation[0..12]: [64, 71, 70, 70, 60, 69, 68, 71, 70, 70, 67, 64]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491834942Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"downstairs","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.49184577Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"downstairs","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"Max found gen is 72","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491855349Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"downstairs","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"Generation requested: 72 >= found:72","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491864917Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"downstairs","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"Next flush: 128","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491874145Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"downstairs","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491883413Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"":"downstairs","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491893302Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.49190206Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"client":"0","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491911708Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"client":"1","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491921186Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"client":"2","":"downstairs","session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491930904Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491939983Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"bc9f7600-804f-47ab-9fe6-ed33c6bef6da is now active with session: c2bc73ff-42e1-4496-820a-8a48b92a2df1","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491949861Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"The guest has finished waiting for activation with:72","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491959939Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491967468Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"session_id":"c2bc73ff-42e1-4496-820a-8a48b92a2df1"} {"msg":"Activated sub_volume bc9f7600-804f-47ab-9fe6-ed33c6bef6da","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491976516Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.491984534Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.492248075Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"task":"crutest"} 01/30 Write block 593 len 24576 data: 10 10 8 6 5 5 02/30 Flush 03/30 Write block 482 len 4096 data: 7 04/30 Write block 214 len 32768 data: 11 11 10 10 10 7 7 6 05/30 Write block 144 len 28672 data: 6 6 4 5 3 3 2 06/30 Write block 27 len 16384 data: 5 6 8 9 07/30 Write block 81 len 36864 data: 5 7 4 7 8 7 5 5 5 08/30 Read block 320 len 36864 09/30 Read block 505 len 24576 10/30 Read block 529 len 8192 11/30 Write block 504 len 4096 data: 10 12/30 Write block 66 len 28672 data: 7 7 6 5 4 3 3 13/30 Write block 72 len 4096 data: 4 14/30 Flush 15/30 Write block 285 len 36864 data: 4 5 5 6 7 7 5 5 5 16/30 Write block 7 len 8192 data: 4 5 17/30 Read block 64 len 28672 18/30 Read block 43 len 28672 19/30 Write block 129 len 12288 data: 10 12 10 20/30 Write block 56 len 36864 data: 9 8 7 8 8 8 9 6 5 21/30 Flush 22/30 Write block 164 len 16384 data: 4 5 6 5 23/30 Write block 71 len 36864 data: 4 5 4 4 5 6 5 6 5 24/30 Read block 234 len 36864 25/30 Write block 375 len 24576 data: 9 9 9 9 8 10 26/30 Flush 27/30 Write block 330 len 16384 data: 8 8 9 7 28/30 Read block 571 len 32768 29/30 Write block 211 len 24576 data: 8 11 10 12 12 11 30/30 Write block 572 len 32768 data: 4 4 3 4 4 4 6 5 ---------------------------------------------------------------- Crucible gen:72 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 1 Done Done Sent false 1003 Acked Write 8 Done Done Sent false 1004 Acked Write 7 Done Done Sent false 1005 Acked Write 4 Done Done Sent false 1006 Acked Write 9 Done Done Sent false 1007 Acked Read 9 Done Done Sent false 1008 Acked Read 6 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Write 1 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 9 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Read 7 Done Done Sent false 1017 Acked Read 7 Done Done Sent false 1018 Acked Write 3 Done Done Sent false 1019 Acked Write 9 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 4 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Read 9 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 4 Done Done Sent false 1027 Acked Read 8 Done Done Sent false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 28 32 Done 26 26 0 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:51.500644593Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:51.500835317Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1421,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 64 130 130 115 F F F <--- 1 020-039 72 72 71 129 129 127 F F F <--- 2 040-059 72 72 70 130 130 125 F F F <--- 3 060-079 72 72 70 131 131 126 F F F <--- 4 080-099 72 72 60 129 129 105 F F F <--- 5 100-119 69 69 69 122 122 122 F F F 6 120-139 72 72 68 130 130 119 F F F <--- 7 140-159 72 72 71 129 129 127 F F F <--- 8 160-179 72 72 70 131 131 123 F F F <--- 9 180-199 70 70 70 126 126 126 F F F 10 200-219 72 72 67 129 129 118 T T F <--- 11 220-239 72 72 64 129 129 112 F F F <--- 12 240-259 69 69 69 122 122 122 F F F 13 260-279 68 68 68 119 119 119 F F F 14 280-299 72 72 64 130 130 112 F F F <--- 15 300-319 70 70 70 126 126 126 F F F 16 320-339 71 71 71 127 127 127 T T F <--- 17 340-359 70 70 70 126 126 126 F F F 18 360-379 72 72 68 131 131 121 F F F <--- 19 380-399 72 72 70 131 131 123 F F F <--- 20 400-419 62 62 62 108 108 108 F F F 21 420-439 67 67 67 118 118 118 F F F 22 440-459 71 71 71 127 127 127 F F F 23 460-479 58 58 58 102 102 102 F F F 24 480-499 72 72 68 129 129 119 F F F <--- 25 500-519 72 72 68 129 129 119 F F F <--- 26 520-539 68 68 68 119 119 119 F F F 27 540-559 70 70 70 124 124 124 F F F 28 560-579 60 60 60 105 105 105 T T F <--- 29 580-599 72 72 72 128 128 128 F F F Max gen: 72, Max flush: 131 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 36 Verifying data now No change after verify Loop: 36 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 130 130 130 F F F 1 020-039 72 72 72 129 129 129 F F F 2 040-059 72 72 72 130 130 130 F F F 3 060-079 72 72 72 131 131 131 F F F 4 080-099 72 72 72 129 129 129 F F F 5 100-119 69 69 69 122 122 122 F F F 6 120-139 72 72 72 130 130 130 F F F 7 140-159 72 72 72 129 129 129 F F F 8 160-179 72 72 72 131 131 131 F F F 9 180-199 70 70 70 126 126 126 F F F 10 200-219 73 73 73 132 132 132 F F F 11 220-239 72 72 72 129 129 129 F F F 12 240-259 69 69 69 122 122 122 F F F 13 260-279 68 68 68 119 119 119 F F F 14 280-299 72 72 72 130 130 130 F F F 15 300-319 70 70 70 126 126 126 F F F 16 320-339 73 73 73 132 132 132 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 72 72 72 131 131 131 F F F 19 380-399 72 72 72 131 131 131 F F F 20 400-419 62 62 62 108 108 108 F F F 21 420-439 67 67 67 118 118 118 F F F 22 440-459 71 71 71 127 127 127 F F F 23 460-479 58 58 58 102 102 102 F F F 24 480-499 72 72 72 129 129 129 F F F 25 500-519 72 72 72 129 129 129 F F F 26 520-539 68 68 68 119 119 119 F F F 27 540-559 70 70 70 124 124 124 F F F 28 560-579 73 73 73 132 132 132 F F F 29 580-599 72 72 72 128 128 128 F F F Max gen: 73, Max flush: 132 Begin loop 37 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.755560923Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.763712357Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.76401493Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764043965Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764052533Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764063911Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429} {"msg":"Crucible b0cfeea8-b85c-46b1-afa2-871d93034771 has session id: 386cbbce-a8b8-4db3-aa30-8f7f5aca6527","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764472375Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"Upstairs opts: Upstairs UUID: b0cfeea8-b85c-46b1-afa2-871d93034771, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.76449741Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"Crucible stats registered with UUID: b0cfeea8-b85c-46b1-afa2-871d93034771","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764507248Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:55.764518426Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764673307Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"io task","client":"1","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764706171Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"io task","client":"0","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764717069Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"io task","client":"2","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764860052Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"task":"crutest"} {"msg":"The guest has requested activation with gen:74","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764875299Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429} {"msg":"b0cfeea8-b85c-46b1-afa2-871d93034771 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764907553Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764917271Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"client":"0","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764927019Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"client":"1","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764936728Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"client":"2","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.764946766Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"io task","client":"2","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.76497547Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"io task","client":"0","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.765017102Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"io task","client":"1","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"ds_connection connected from Ok(127.0.0.1:48063)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.765041238Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"io task","client":"1","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"ds_connection connected from Ok(127.0.0.1:51829)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.765052496Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"io task","client":"0","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"ds_connection connected from Ok(127.0.0.1:38230)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.765063034Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"io task","client":"2","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.765769172Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"client":"0","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.765805245Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"client":"1","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.765819392Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"client":"2","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.76610062Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766132394Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"[0]R flush_numbers[0..12]: [130, 129, 130, 131, 129, 122, 130, 129, 131, 126, 132, 129]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766188283Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"downstairs","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"[0]R generation[0..12]: [72, 72, 72, 72, 72, 69, 72, 72, 72, 70, 73, 72]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766199321Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"downstairs","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766210429Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"downstairs","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"[1]R flush_numbers[0..12]: [130, 129, 130, 131, 129, 122, 130, 129, 131, 126, 132, 129]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766221037Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"downstairs","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"[1]R generation[0..12]: [72, 72, 72, 72, 72, 69, 72, 72, 72, 70, 73, 72]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766231275Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"downstairs","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766242123Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"downstairs","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"[2]R flush_numbers[0..12]: [130, 129, 130, 131, 129, 122, 130, 129, 131, 126, 132, 129]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766252691Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"downstairs","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"[2]R generation[0..12]: [72, 72, 72, 72, 72, 69, 72, 72, 72, 70, 73, 72]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766262989Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"downstairs","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766273847Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"downstairs","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"Max found gen is 74","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766284865Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"downstairs","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"Generation requested: 74 >= found:74","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766294393Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"downstairs","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"Next flush: 133","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766303722Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"downstairs","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.76631312Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"":"downstairs","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766322758Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766331666Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"client":"0","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766341334Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"client":"1","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766350873Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"client":"2","":"downstairs","session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766360761Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766369769Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"b0cfeea8-b85c-46b1-afa2-871d93034771 is now active with session: 386cbbce-a8b8-4db3-aa30-8f7f5aca6527","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766379567Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766388456Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"session_id":"386cbbce-a8b8-4db3-aa30-8f7f5aca6527"} {"msg":"The guest has finished waiting for activation with:74","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766397464Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429} {"msg":"Activated sub_volume b0cfeea8-b85c-46b1-afa2-871d93034771","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766405323Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766413441Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.766732301Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"task":"crutest"} 01/30 Write block 411 len 24576 data: 3 3 2 4 4 4 02/30 Flush 03/30 Write block 228 len 12288 data: 5 4 5 04/30 Flush 05/30 Write block 22 len 16384 data: 5 6 6 6 06/30 Read block 374 len 20480 07/30 Read block 63 len 28672 08/30 Write block 470 len 40960 data: 7 7 5 4 3 4 3 4 4 5 09/30 Flush 10/30 Write block 316 len 12288 data: 8 9 7 11/30 Read block 410 len 16384 12/30 Flush 13/30 Write block 324 len 16384 data: 7 7 8 7 14/30 Read block 208 len 20480 15/30 Write block 362 len 24576 data: 2 2 6 6 5 3 16/30 Write block 539 len 28672 data: 4 4 4 5 7 7 7 17/30 Read block 527 len 40960 18/30 Read block 28 len 40960 19/30 Write block 191 len 24576 data: 9 10 12 9 10 10 20/30 Read block 306 len 28672 21/30 Flush 22/30 Write block 518 len 36864 data: 5 4 4 5 5 5 8 8 8 23/30 Read block 488 len 4096 24/30 Read block 18 len 36864 25/30 Read block 261 len 36864 26/30 Read block 323 len 32768 27/30 Write block 402 len 12288 data: 6 6 6 28/30 Write block 365 len 4096 data: 7 29/30 Write block 519 len 28672 data: 5 5 6 6 6 9 9 30/30 Write block 126 len 40960 data: 12 11 10 11 13 11 10 10 8 7 ---------------------------------------------------------------- Crucible gen:74 GIO:true work queues: Upstairs:0 downstairs:26 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1004 Acked Write 4 Done Done Done false 1005 Acked Read 5 Done Done Done false 1006 Acked Read 7 Done Done Done false 1007 Acked Write 10 Done Done Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Read 4 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 4 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Write 6 Done Sent Done false 1015 Acked Write 7 Done Sent Done false 1016 Acked Read 10 Done Sent Done false 1017 Acked Read 10 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Read 7 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 9 Done Sent Done false 1022 Acked Read 1 Done Sent Done false 1023 Acked Read 9 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Write 3 Done Sent Done false 1027 Acked Write 1 Done Sent Done false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 22 2 26 Done 24 4 24 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1003)) Some(JobId(1020)) Downstairs last five completed: 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:58:55.776183131Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:58:55.776376915Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1429,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 130 130 130 F F F 1 020-039 74 72 74 135 129 135 F T F <--- 2 040-059 72 72 72 130 130 130 F F F 3 060-079 72 72 72 131 131 131 F F F 4 080-099 72 72 72 129 129 129 F F F 5 100-119 69 69 69 122 122 122 F F F 6 120-139 72 72 72 130 130 130 T F T <--- 7 140-159 72 72 72 129 129 129 F F F 8 160-179 72 72 72 131 131 131 F F F 9 180-199 74 70 74 137 126 137 F F F <--- 10 200-219 73 73 73 132 132 132 F F F 11 220-239 74 74 74 134 134 134 F F F 12 240-259 69 69 69 122 122 122 F F F 13 260-279 68 68 68 119 119 119 F F F 14 280-299 72 72 72 130 130 130 F F F 15 300-319 74 70 74 136 126 136 F F F <--- 16 320-339 74 73 74 137 132 137 F F F <--- 17 340-359 70 70 70 126 126 126 F F F 18 360-379 74 72 74 137 131 137 T F T <--- 19 380-399 72 72 72 131 131 131 F F F 20 400-419 74 74 74 133 133 133 T F T <--- 21 420-439 67 67 67 118 118 118 F F F 22 440-459 71 71 71 127 127 127 F F F 23 460-479 74 58 74 135 102 135 F T F <--- 24 480-499 72 72 72 129 129 129 F F F 25 500-519 72 72 72 129 129 129 T F T <--- 26 520-539 74 68 74 137 119 137 T F T <--- 27 540-559 74 70 74 137 124 137 F F F <--- 28 560-579 73 73 73 132 132 132 F F F 29 580-599 72 72 72 128 128 128 F F F Max gen: 74, Max flush: 137 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 37 Verifying data now No change after verify Loop: 37 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 130 130 130 F F F 1 020-039 74 74 74 135 135 135 F F F 2 040-059 72 72 72 130 130 130 F F F 3 060-079 72 72 72 131 131 131 F F F 4 080-099 72 72 72 129 129 129 F F F 5 100-119 69 69 69 122 122 122 F F F 6 120-139 75 75 75 138 138 138 F F F 7 140-159 72 72 72 129 129 129 F F F 8 160-179 72 72 72 131 131 131 F F F 9 180-199 74 74 74 137 137 137 F F F 10 200-219 73 73 73 132 132 132 F F F 11 220-239 74 74 74 134 134 134 F F F 12 240-259 69 69 69 122 122 122 F F F 13 260-279 68 68 68 119 119 119 F F F 14 280-299 72 72 72 130 130 130 F F F 15 300-319 74 74 74 136 136 136 F F F 16 320-339 74 74 74 137 137 137 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 75 75 75 138 138 138 F F F 19 380-399 72 72 72 131 131 131 F F F 20 400-419 75 75 75 138 138 138 F F F 21 420-439 67 67 67 118 118 118 F F F 22 440-459 71 71 71 127 127 127 F F F 23 460-479 74 74 74 135 135 135 F F F 24 480-499 72 72 72 129 129 129 F F F 25 500-519 75 75 75 138 138 138 F F F 26 520-539 75 75 75 138 138 138 F F F 27 540-559 74 74 74 137 137 137 F F F 28 560-579 73 73 73 132 132 132 F F F 29 580-599 72 72 72 128 128 128 F F F Max gen: 75, Max flush: 138 Begin loop 38 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.02614477Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.034250832Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.034548996Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.034575981Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.034584399Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.034595787Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437} {"msg":"Crucible 73e2de85-cc73-45f0-8226-1d10bbb8eb07 has session id: 553ad2d8-ae8a-4285-8a3f-55a75476a98f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.034980765Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"Upstairs opts: Upstairs UUID: 73e2de85-cc73-45f0-8226-1d10bbb8eb07, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.03500915Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"Crucible stats registered with UUID: 73e2de85-cc73-45f0-8226-1d10bbb8eb07","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035020367Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:00.035029556Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.03516726Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"io task","client":"1","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035184357Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"io task","client":"0","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035194335Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"io task","client":"2","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035330849Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"task":"crutest"} {"msg":"The guest has requested activation with gen:76","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035348266Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437} {"msg":"73e2de85-cc73-45f0-8226-1d10bbb8eb07 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035383429Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035395157Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"client":"0","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035404945Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"client":"1","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035414544Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"client":"2","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035424042Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"io task","client":"2","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.03543393Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"io task","client":"0","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035443838Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"io task","client":"1","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"ds_connection connected from Ok(127.0.0.1:54210)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035495888Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"io task","client":"0","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"ds_connection connected from Ok(127.0.0.1:40059)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035506976Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"io task","client":"2","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"ds_connection connected from Ok(127.0.0.1:59861)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.035517244Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"io task","client":"1","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.03618267Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"client":"0","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036196677Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"client":"1","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036293779Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"client":"2","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036482354Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.03649942Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"[0]R flush_numbers[0..12]: [130, 135, 130, 131, 129, 122, 138, 129, 131, 137, 132, 134]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.03655654Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"downstairs","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"[0]R generation[0..12]: [72, 74, 72, 72, 72, 69, 75, 72, 72, 74, 73, 74]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036568987Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"downstairs","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036579745Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"downstairs","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"[1]R flush_numbers[0..12]: [130, 135, 130, 131, 129, 122, 138, 129, 131, 137, 132, 134]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036590083Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"downstairs","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"[1]R generation[0..12]: [72, 74, 72, 72, 72, 69, 75, 72, 72, 74, 73, 74]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036600592Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"downstairs","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036611369Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"downstairs","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"[2]R flush_numbers[0..12]: [130, 135, 130, 131, 129, 122, 138, 129, 131, 137, 132, 134]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036621698Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"downstairs","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"[2]R generation[0..12]: [72, 74, 72, 72, 72, 69, 75, 72, 72, 74, 73, 74]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036631726Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"downstairs","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036642494Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"downstairs","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"Max found gen is 76","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036651802Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"downstairs","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"Generation requested: 76 >= found:76","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.03666147Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"downstairs","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"Next flush: 139","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036670628Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"downstairs","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036679937Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"":"downstairs","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036689865Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036698603Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"client":"0","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036709311Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"client":"1","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036719089Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"client":"2","":"downstairs","session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036728907Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036738076Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"73e2de85-cc73-45f0-8226-1d10bbb8eb07 is now active with session: 553ad2d8-ae8a-4285-8a3f-55a75476a98f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036747874Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036756462Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"session_id":"553ad2d8-ae8a-4285-8a3f-55a75476a98f"} {"msg":"The guest has finished waiting for activation with:76","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036765461Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437} {"msg":"Activated sub_volume 73e2de85-cc73-45f0-8226-1d10bbb8eb07","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036773379Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.036781338Z"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"task":"crutest"} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.037046258Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"task":"crutest"} 01/30 Flush 02/30 Write block 237 len 16384 data: 4 4 2 3 03/30 Write block 573 len 36864 data: 5 4 5 5 5 7 6 5 5 04/30 Write block 591 len 28672 data: 9 9 11 11 9 7 6 05/30 Read block 12 len 8192 06/30 Write block 369 len 28672 data: 3 4 7 7 6 7 10 07/30 Write block 99 len 16384 data: 2 6 5 5 08/30 Write block 445 len 12288 data: 6 4 3 09/30 Flush 10/30 Write block 521 len 20480 data: 7 7 7 10 10 11/30 Write block 541 len 4096 data: 5 12/30 Write block 450 len 28672 data: 3 4 3 4 5 4 7 13/30 Write block 390 len 20480 data: 6 7 8 8 7 14/30 Write block 587 len 12288 data: 8 9 10 15/30 Read block 88 len 4096 16/30 Write block 546 len 32768 data: 6 5 5 4 4 4 5 5 17/30 Flush 18/30 Write block 161 len 8192 data: 3 4 19/30 Write block 332 len 32768 data: 10 8 8 8 8 6 7 6 20/30 Write block 581 len 16384 data: 6 6 5 4 21/30 Write block 320 len 8192 data: 9 9 22/30 Read block 332 len 12288 23/30 Read block 135 len 12288 24/30 Flush 25/30 Write block 422 len 36864 data: 4 4 4 4 4 5 5 5 5 26/30 Write block 535 len 16384 data: 5 4 4 4 27/30 Read block 54 len 12288 28/30 Write block 137 len 16384 data: 6 8 8 8 29/30 Write block 292 len 8192 data: 6 6 30/30 Write block 91 len 4096 data: 5 ---------------------------------------------------------------- Crucible gen:76 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 4 Done Done Done false 1002 Acked Write 9 Done Done Done false 1003 Acked Write 7 Done Done Done false 1004 Acked Read 2 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Write 4 Done Done Sent false 1007 Acked Write 3 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 5 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Write 7 Done Done Sent false 1012 Acked Write 5 Done Done Sent false 1013 Acked Write 3 Done Done Sent false 1014 Acked Read 1 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 2 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Write 4 Done Done Sent false 1020 Acked Write 2 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Read 3 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 9 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Write 4 Sent Sent Sent false 1028 Acked Write 2 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 26 32 Done 26 26 3 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) Some(JobId(1000)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:00.044368057Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:00.04456406Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1437,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 130 130 130 F F F 1 020-039 74 74 74 135 135 135 F F F 2 040-059 72 72 72 130 130 130 F F F 3 060-079 72 72 72 131 131 131 F F F 4 080-099 76 76 72 140 140 129 T T F <--- 5 100-119 76 76 69 140 140 122 F F F <--- 6 120-139 75 75 75 138 138 138 T T F <--- 7 140-159 72 72 72 129 129 129 T T F <--- 8 160-179 76 76 72 142 142 131 F F F <--- 9 180-199 74 74 74 137 137 137 F F F 10 200-219 73 73 73 132 132 132 F F F 11 220-239 76 76 74 140 140 134 F F T <--- 12 240-259 76 76 69 140 140 122 F F T <--- 13 260-279 68 68 68 119 119 119 F F F 14 280-299 72 72 72 130 130 130 T T F <--- 15 300-319 74 74 74 136 136 136 F F F 16 320-339 76 76 74 142 142 137 F F F <--- 17 340-359 70 70 70 126 126 126 F F F 18 360-379 76 76 75 140 140 138 F F F <--- 19 380-399 76 76 72 141 141 131 F F F <--- 20 400-419 75 75 75 138 138 138 F F F 21 420-439 67 67 67 118 118 118 T T F <--- 22 440-459 76 76 71 141 141 127 F F F <--- 23 460-479 74 74 74 135 135 135 F F F 24 480-499 72 72 72 129 129 129 F F F 25 500-519 75 75 75 138 138 138 F F F 26 520-539 76 76 75 141 141 138 T T F <--- 27 540-559 76 76 74 141 141 137 F F F <--- 28 560-579 76 76 73 140 140 132 F F T <--- 29 580-599 76 76 72 142 142 128 F F T <--- Max gen: 76, Max flush: 142 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 38 Verifying data now No change after verify Loop: 38 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 72 72 130 130 130 F F F 1 020-039 74 74 74 135 135 135 F F F 2 040-059 72 72 72 130 130 130 F F F 3 060-079 72 72 72 131 131 131 F F F 4 080-099 77 77 77 143 143 143 F F F 5 100-119 76 76 76 140 140 140 F F F 6 120-139 77 77 77 143 143 143 F F F 7 140-159 77 77 77 143 143 143 F F F 8 160-179 76 76 76 142 142 142 F F F 9 180-199 74 74 74 137 137 137 F F F 10 200-219 73 73 73 132 132 132 F F F 11 220-239 76 76 76 140 140 140 F F F 12 240-259 76 76 76 140 140 140 F F F 13 260-279 68 68 68 119 119 119 F F F 14 280-299 77 77 77 143 143 143 F F F 15 300-319 74 74 74 136 136 136 F F F 16 320-339 76 76 76 142 142 142 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 76 76 76 140 140 140 F F F 19 380-399 76 76 76 141 141 141 F F F 20 400-419 75 75 75 138 138 138 F F F 21 420-439 77 77 77 143 143 143 F F F 22 440-459 76 76 76 141 141 141 F F F 23 460-479 74 74 74 135 135 135 F F F 24 480-499 72 72 72 129 129 129 F F F 25 500-519 75 75 75 138 138 138 F F F 26 520-539 77 77 77 143 143 143 F F F 27 540-559 76 76 76 141 141 141 F F F 28 560-579 76 76 76 140 140 140 F F F 29 580-599 76 76 76 142 142 142 F F F Max gen: 77, Max flush: 143 Begin loop 39 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.309478568Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:04.316798508Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.316849918Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.323818793Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324093861Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324124666Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324135453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324146961Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445} {"msg":"Crucible 2a8d72c3-4019-4b7d-8d73-767f3bb1b58e has session id: 7f4c7f98-18a9-4d32-915c-0efce34e3234","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324503755Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"Upstairs opts: Upstairs UUID: 2a8d72c3-4019-4b7d-8d73-767f3bb1b58e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324534999Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"Crucible stats registered with UUID: 2a8d72c3-4019-4b7d-8d73-767f3bb1b58e","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324545337Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:04.324554315Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324713245Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"io task","client":"1","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.32474355Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"io task","client":"0","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324754768Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"io task","client":"2","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.32489918Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"task":"crutest"} {"msg":"The guest has requested activation with gen:78","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324915147Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445} {"msg":"2a8d72c3-4019-4b7d-8d73-767f3bb1b58e active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324947281Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.32495699Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"client":"0","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324966668Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"client":"1","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324978176Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"client":"2","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324987734Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"io task","client":"2","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.324997722Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"io task","client":"0","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.32500774Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"io task","client":"1","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"ds_connection connected from Ok(127.0.0.1:42136)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.325074438Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"io task","client":"1","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"ds_connection connected from Ok(127.0.0.1:60986)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.325085526Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"io task","client":"0","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"ds_connection connected from Ok(127.0.0.1:49482)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.325096494Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"io task","client":"2","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.325893674Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"client":"2","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.325910801Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"client":"1","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.325986287Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"client":"0","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326219433Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326242129Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"[0]R flush_numbers[0..12]: [130, 135, 130, 131, 143, 140, 143, 143, 142, 137, 132, 140]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326260376Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"downstairs","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"[0]R generation[0..12]: [72, 74, 72, 72, 77, 76, 77, 77, 76, 74, 73, 76]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326271353Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"downstairs","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326282341Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"downstairs","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"[1]R flush_numbers[0..12]: [130, 135, 130, 131, 143, 140, 143, 143, 142, 137, 132, 140]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326295519Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"downstairs","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"[1]R generation[0..12]: [72, 74, 72, 72, 77, 76, 77, 77, 76, 74, 73, 76]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326305847Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"downstairs","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326316735Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"downstairs","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"[2]R flush_numbers[0..12]: [130, 135, 130, 131, 143, 140, 143, 143, 142, 137, 132, 140]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326327263Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"downstairs","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"[2]R generation[0..12]: [72, 74, 72, 72, 77, 76, 77, 77, 76, 74, 73, 76]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326337771Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"downstairs","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326348609Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"downstairs","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"Max found gen is 78","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326357897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"downstairs","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"Generation requested: 78 >= found:78","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326367435Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"downstairs","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"Next flush: 144","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326376744Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"downstairs","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326386212Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"":"downstairs","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.32639593Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326405058Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"client":"0","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326414737Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"client":"1","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326424425Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"client":"2","":"downstairs","session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326434203Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326444931Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"2a8d72c3-4019-4b7d-8d73-767f3bb1b58e is now active with session: 7f4c7f98-18a9-4d32-915c-0efce34e3234","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326454799Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326468826Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"session_id":"7f4c7f98-18a9-4d32-915c-0efce34e3234"} {"msg":"The guest has finished waiting for activation with:78","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326478015Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445} {"msg":"Activated sub_volume 2a8d72c3-4019-4b7d-8d73-767f3bb1b58e","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326486293Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326494602Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.326828599Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"task":"crutest"} 01/30 Write block 237 len 28672 data: 5 5 3 4 6 6 6 02/30 Write block 97 len 12288 data: 4 2 3 03/30 Write block 232 len 4096 data: 5 04/30 Read block 310 len 28672 05/30 Read block 39 len 32768 06/30 Read block 107 len 40960 07/30 Read block 311 len 8192 08/30 Write block 551 len 24576 data: 5 6 6 6 6 6 09/30 Read block 575 len 28672 10/30 Flush 11/30 Write block 19 len 20480 data: 9 8 6 6 7 12/30 Write block 507 len 32768 data: 7 3 4 4 4 4 4 5 13/30 Read block 578 len 8192 14/30 Read block 136 len 32768 15/30 Write block 258 len 20480 data: 7 6 6 6 6 16/30 Write block 107 len 32768 data: 8 6 7 8 8 8 7 7 17/30 Read block 214 len 28672 18/30 Flush 19/30 Write block 54 len 36864 data: 6 7 10 9 8 9 9 9 10 20/30 Read block 372 len 8192 21/30 Read block 295 len 4096 22/30 Write block 396 len 12288 data: 7 8 7 23/30 Flush 24/30 Write block 449 len 16384 data: 3 4 5 4 25/30 Read block 517 len 20480 26/30 Read block 51 len 12288 27/30 Read block 431 len 28672 28/30 Write block 106 len 36864 data: 6 9 7 8 9 9 9 8 8 29/30 Read block 527 len 28672 30/30 Read block 37 len 36864 ---------------------------------------------------------------- Crucible gen:78 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 3 Sent Done Done false 1002 Acked Write 1 Sent Done Done false 1003 Acked Read 7 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Read 10 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Flush 0 Sent Done Done false 1010 Acked Write 5 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Read 2 Sent Done Done false 1013 Acked Read 8 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Write 8 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Read 1 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 4 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Read 7 Sent Done Done false 1027 Acked Write 9 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Read 9 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 0 30 Done 1 29 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:04.336574174Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:04.336758139Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1445,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 72 78 78 130 145 145 F F F <--- 1 020-039 74 78 78 135 145 145 F F F <--- 2 040-059 72 78 78 130 146 146 F F F <--- 3 060-079 72 78 78 131 146 146 F F F <--- 4 080-099 77 78 78 143 144 144 F F F <--- 5 100-119 76 78 78 140 145 145 F T T <--- 6 120-139 77 77 77 143 143 143 F F F 7 140-159 77 77 77 143 143 143 F F F 8 160-179 76 76 76 142 142 142 F F F 9 180-199 74 74 74 137 137 137 F F F 10 200-219 73 73 73 132 132 132 F F F 11 220-239 76 78 78 140 144 144 T F F <--- 12 240-259 76 78 78 140 145 145 T F F <--- 13 260-279 68 78 78 119 145 145 F F F <--- 14 280-299 77 77 77 143 143 143 F F F 15 300-319 74 74 74 136 136 136 F F F 16 320-339 76 76 76 142 142 142 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 76 76 76 140 140 140 F F F 19 380-399 76 78 78 141 146 146 F F F <--- 20 400-419 75 75 75 138 138 138 F F F 21 420-439 77 77 77 143 143 143 F F F 22 440-459 76 76 76 141 141 141 F T T <--- 23 460-479 74 74 74 135 135 135 F F F 24 480-499 72 72 72 129 129 129 F F F 25 500-519 75 78 78 138 145 145 F F F <--- 26 520-539 77 77 77 143 143 143 F F F 27 540-559 76 78 78 141 144 144 F F F <--- 28 560-579 76 76 76 140 140 140 F F F 29 580-599 76 76 76 142 142 142 F F F Max gen: 78, Max flush: 146 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 39 Verifying data now No change after verify Loop: 39 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 78 78 78 145 145 145 F F F 1 020-039 78 78 78 145 145 145 F F F 2 040-059 78 78 78 146 146 146 F F F 3 060-079 78 78 78 146 146 146 F F F 4 080-099 78 78 78 144 144 144 F F F 5 100-119 79 79 79 147 147 147 F F F 6 120-139 77 77 77 143 143 143 F F F 7 140-159 77 77 77 143 143 143 F F F 8 160-179 76 76 76 142 142 142 F F F 9 180-199 74 74 74 137 137 137 F F F 10 200-219 73 73 73 132 132 132 F F F 11 220-239 78 78 78 144 144 144 F F F 12 240-259 78 78 78 145 145 145 F F F 13 260-279 78 78 78 145 145 145 F F F 14 280-299 77 77 77 143 143 143 F F F 15 300-319 74 74 74 136 136 136 F F F 16 320-339 76 76 76 142 142 142 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 76 76 76 140 140 140 F F F 19 380-399 78 78 78 146 146 146 F F F 20 400-419 75 75 75 138 138 138 F F F 21 420-439 77 77 77 143 143 143 F F F 22 440-459 79 79 79 147 147 147 F F F 23 460-479 74 74 74 135 135 135 F F F 24 480-499 72 72 72 129 129 129 F F F 25 500-519 78 78 78 145 145 145 F F F 26 520-539 77 77 77 143 143 143 F F F 27 540-559 78 78 78 144 144 144 F F F 28 560-579 76 76 76 140 140 140 F F F 29 580-599 76 76 76 142 142 142 F F F Max gen: 79, Max flush: 147 Begin loop 40 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.59317319Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.601580475Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.601909054Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.601934899Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.601943567Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.601954975Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453} {"msg":"Crucible b609bb63-fb97-4f86-8cc6-53cd63beeea1 has session id: 883cbed3-758e-4f13-895d-e455d804632c","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.6023546Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"Upstairs opts: Upstairs UUID: b609bb63-fb97-4f86-8cc6-53cd63beeea1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602378866Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"Crucible stats registered with UUID: b609bb63-fb97-4f86-8cc6-53cd63beeea1","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602390743Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:08.602399812Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602559202Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"io task","client":"1","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602588636Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"io task","client":"0","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602599444Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"io task","client":"2","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602704645Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"task":"crutest"} {"msg":"The guest has requested activation with gen:80","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602733259Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453} {"msg":"b609bb63-fb97-4f86-8cc6-53cd63beeea1 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602745357Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602754105Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"client":"0","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602763914Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"client":"1","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602773652Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"client":"2","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.60278335Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"io task","client":"2","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602793588Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"io task","client":"0","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602803666Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"io task","client":"1","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"ds_connection connected from Ok(127.0.0.1:59626)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602862965Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"io task","client":"2","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"ds_connection connected from Ok(127.0.0.1:65450)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.60289038Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"io task","client":"0","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"ds_connection connected from Ok(127.0.0.1:37872)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.602973044Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"io task","client":"1","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.603514903Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"client":"1","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.603555305Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"client":"0","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.60358472Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"client":"2","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.603827994Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.603863228Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"[0]R flush_numbers[0..12]: [145, 145, 146, 146, 144, 147, 143, 143, 142, 137, 132, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.60390206Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"downstairs","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"[0]R generation[0..12]: [78, 78, 78, 78, 78, 79, 77, 77, 76, 74, 73, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.603913228Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"downstairs","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.603924316Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"downstairs","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"[1]R flush_numbers[0..12]: [145, 145, 146, 146, 144, 147, 143, 143, 142, 137, 132, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.603934854Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"downstairs","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"[1]R generation[0..12]: [78, 78, 78, 78, 78, 79, 77, 77, 76, 74, 73, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.603945312Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"downstairs","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.60395622Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"downstairs","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"[2]R flush_numbers[0..12]: [145, 145, 146, 146, 144, 147, 143, 143, 142, 137, 132, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.603966878Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"downstairs","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"[2]R generation[0..12]: [78, 78, 78, 78, 78, 79, 77, 77, 76, 74, 73, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.603979036Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"downstairs","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.603989844Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"downstairs","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"Max found gen is 80","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.603999342Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"downstairs","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"Generation requested: 80 >= found:80","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.60400899Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"downstairs","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"Next flush: 148","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.604018339Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"downstairs","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.604027527Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"":"downstairs","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.604037285Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.604046083Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"client":"0","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.604055732Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"client":"1","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.60406527Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"client":"2","":"downstairs","session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.604074978Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.604084126Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"b609bb63-fb97-4f86-8cc6-53cd63beeea1 is now active with session: 883cbed3-758e-4f13-895d-e455d804632c","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.604093984Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.604102633Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"session_id":"883cbed3-758e-4f13-895d-e455d804632c"} {"msg":"The guest has finished waiting for activation with:80","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.604111731Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453} {"msg":"Activated sub_volume b609bb63-fb97-4f86-8cc6-53cd63beeea1","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.60411995Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.604128008Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.604427602Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"task":"crutest"} 01/30 Write block 119 len 40960 data: 6 7 8 7 9 11 11 13 12 11 02/30 Read block 352 len 16384 03/30 Read block 155 len 28672 04/30 Write block 553 len 24576 data: 7 7 7 7 10 9 05/30 Read block 100 len 32768 06/30 Read block 199 len 20480 07/30 Read block 502 len 20480 08/30 Write block 258 len 28672 data: 8 7 7 7 7 6 4 09/30 Flush 10/30 Write block 139 len 12288 data: 9 9 8 11/30 Read block 72 len 20480 12/30 Write block 404 len 20480 data: 7 5 5 5 4 13/30 Read block 395 len 28672 14/30 Read block 328 len 20480 15/30 Read block 421 len 32768 16/30 Read block 260 len 24576 17/30 Read block 152 len 4096 18/30 Read block 149 len 8192 19/30 Write block 528 len 28672 data: 8 8 5 5 5 5 5 20/30 Read block 50 len 24576 21/30 Read block 255 len 8192 22/30 Read block 215 len 8192 23/30 Read block 192 len 40960 24/30 Write block 198 len 20480 data: 5 4 3 3 5 25/30 Write block 551 len 28672 data: 6 7 8 8 8 8 11 26/30 Write block 59 len 24576 data: 10 10 10 11 7 6 27/30 Read block 397 len 4096 28/30 Write block 300 len 16384 data: 7 7 9 11 29/30 Read block 483 len 24576 30/30 Read block 436 len 4096 ---------------------------------------------------------------- Crucible gen:80 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Read 7 Done Done Done false 1003 Acked Write 6 Done Done Done false 1004 Acked Read 8 Done Done Done false 1005 Acked Read 5 Done Done Done false 1006 Acked Read 5 Done Done Done false 1007 Acked Write 7 Done Done Sent false 1008 Acked Flush 0 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Read 5 Done Done Sent false 1011 Acked Write 5 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Read 5 Done Done Sent false 1014 Acked Read 8 Done Done Sent false 1015 Acked Read 6 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Write 7 Done Done Sent false 1019 Acked Read 6 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Read 10 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Write 7 Done Done Sent false 1025 Acked Write 6 Done Done Sent false 1026 Acked Read 1 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Read 1 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 23 24 Done 30 29 7 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1008)) Some(JobId(1008)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:08.613812244Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:08.61399341Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1453,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 78 78 78 145 145 145 F F F 1 020-039 78 78 78 145 145 145 F F F 2 040-059 78 78 78 146 146 146 T T F <--- 3 060-079 78 78 78 146 146 146 T T F <--- 4 080-099 78 78 78 144 144 144 F F F 5 100-119 80 80 79 148 148 147 F F T <--- 6 120-139 80 80 77 148 148 143 T T T <--- 7 140-159 77 77 77 143 143 143 T T F <--- 8 160-179 76 76 76 142 142 142 F F F 9 180-199 74 74 74 137 137 137 T T F <--- 10 200-219 73 73 73 132 132 132 T T F <--- 11 220-239 78 78 78 144 144 144 F F F 12 240-259 80 80 78 148 148 145 F F F <--- 13 260-279 80 80 78 148 148 145 F F F <--- 14 280-299 77 77 77 143 143 143 F F F 15 300-319 74 74 74 136 136 136 T T F <--- 16 320-339 76 76 76 142 142 142 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 76 76 76 140 140 140 F F F 19 380-399 78 78 78 146 146 146 F F F 20 400-419 75 75 75 138 138 138 T T F <--- 21 420-439 77 77 77 143 143 143 F F F 22 440-459 79 79 79 147 147 147 F F F 23 460-479 74 74 74 135 135 135 F F F 24 480-499 72 72 72 129 129 129 F F F 25 500-519 78 78 78 145 145 145 F F F 26 520-539 77 77 77 143 143 143 T T F <--- 27 540-559 80 80 78 148 148 144 T T T <--- 28 560-579 76 76 76 140 140 140 F F F 29 580-599 76 76 76 142 142 142 F F F Max gen: 80, Max flush: 148 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 40 Verifying data now No change after verify Loop: 40 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 78 78 78 145 145 145 F F F 1 020-039 78 78 78 145 145 145 F F F 2 040-059 81 81 81 149 149 149 F F F 3 060-079 81 81 81 149 149 149 F F F 4 080-099 78 78 78 144 144 144 F F F 5 100-119 80 80 80 148 148 148 F F F 6 120-139 81 81 81 149 149 149 F F F 7 140-159 81 81 81 149 149 149 F F F 8 160-179 76 76 76 142 142 142 F F F 9 180-199 81 81 81 149 149 149 F F F 10 200-219 81 81 81 149 149 149 F F F 11 220-239 78 78 78 144 144 144 F F F 12 240-259 80 80 80 148 148 148 F F F 13 260-279 80 80 80 148 148 148 F F F 14 280-299 77 77 77 143 143 143 F F F 15 300-319 81 81 81 149 149 149 F F F 16 320-339 76 76 76 142 142 142 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 76 76 76 140 140 140 F F F 19 380-399 78 78 78 146 146 146 F F F 20 400-419 81 81 81 149 149 149 F F F 21 420-439 77 77 77 143 143 143 F F F 22 440-459 79 79 79 147 147 147 F F F 23 460-479 74 74 74 135 135 135 F F F 24 480-499 72 72 72 129 129 129 F F F 25 500-519 78 78 78 145 145 145 F F F 26 520-539 81 81 81 149 149 149 F F F 27 540-559 81 81 81 149 149 149 F F F 28 560-579 76 76 76 140 140 140 F F F 29 580-599 76 76 76 142 142 142 F F F Max gen: 81, Max flush: 149 Begin loop 41 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.884703073Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.892970625Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.893271018Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.893298133Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.893306872Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.89331841Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461} {"msg":"Crucible 2405c633-07f1-4ab7-8734-535631c4513c has session id: 14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.893645628Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"Upstairs opts: Upstairs UUID: 2405c633-07f1-4ab7-8734-535631c4513c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.893686511Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"Crucible stats registered with UUID: 2405c633-07f1-4ab7-8734-535631c4513c","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.893697249Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:12.893706297Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.8938526Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"io task","client":"0","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.893869746Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"io task","client":"1","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.893882164Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"io task","client":"2","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.894051082Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"task":"crutest"} {"msg":"The guest has requested activation with gen:82","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.894067809Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461} {"msg":"2405c633-07f1-4ab7-8734-535631c4513c active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.894085286Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.894094134Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"client":"0","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.894103752Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"client":"1","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.894113331Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"client":"2","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.894122859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"io task","client":"2","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.894132837Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"io task","client":"0","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.894142835Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"io task","client":"1","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:57747)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.894187667Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"io task","client":"0","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:42615)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.894213072Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"io task","client":"2","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"ds_connection connected from Ok(127.0.0.1:33180)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.894245786Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"io task","client":"1","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.894968001Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"client":"0","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.89502348Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"client":"2","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895035928Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"client":"1","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895349409Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895393781Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"[0]R flush_numbers[0..12]: [145, 145, 149, 149, 144, 148, 149, 149, 142, 149, 149, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895455689Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"downstairs","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"[0]R generation[0..12]: [78, 78, 81, 81, 78, 80, 81, 81, 76, 81, 81, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895467007Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"downstairs","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895478075Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"downstairs","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"[1]R flush_numbers[0..12]: [145, 145, 149, 149, 144, 148, 149, 149, 142, 149, 149, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895488713Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"downstairs","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"[1]R generation[0..12]: [78, 78, 81, 81, 78, 80, 81, 81, 76, 81, 81, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895499011Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"downstairs","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895509879Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"downstairs","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"[2]R flush_numbers[0..12]: [145, 145, 149, 149, 144, 148, 149, 149, 142, 149, 149, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895520367Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"downstairs","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"[2]R generation[0..12]: [78, 78, 81, 81, 78, 80, 81, 81, 76, 81, 81, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895530665Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"downstairs","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895541363Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"downstairs","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"Max found gen is 82","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895550781Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"downstairs","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"Generation requested: 82 >= found:82","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.89556046Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"downstairs","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"Next flush: 150","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895569908Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"downstairs","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895580756Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"":"downstairs","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895590754Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895599632Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"client":"0","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.89560956Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"client":"1","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895619259Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"client":"2","":"downstairs","session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895629087Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895638045Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"2405c633-07f1-4ab7-8734-535631c4513c is now active with session: 14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895647913Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895656522Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"session_id":"14ed3ba1-6aff-4dff-8d89-8ae1ec2dc5bb"} {"msg":"The guest has finished waiting for activation with:82","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.89566565Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461} {"msg":"Activated sub_volume 2405c633-07f1-4ab7-8734-535631c4513c","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895673598Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.895681657Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.896033871Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"task":"crutest"} 01/30 Write block 93 len 16384 data: 5 6 5 5 02/30 Write block 6 len 32768 data: 4 5 6 5 8 10 10 8 03/30 Read block 452 len 4096 04/30 Read block 455 len 28672 05/30 Read block 181 len 20480 06/30 Read block 183 len 4096 07/30 Write block 132 len 36864 data: 11 11 9 8 6 7 9 10 10 08/30 Read block 188 len 20480 09/30 Read block 387 len 36864 10/30 Read block 566 len 40960 11/30 Read block 360 len 40960 12/30 Read block 395 len 20480 13/30 Write block 590 len 28672 data: 11 10 10 12 12 10 8 14/30 Read block 462 len 20480 15/30 Read block 589 len 16384 16/30 Read block 33 len 28672 17/30 Read block 271 len 32768 18/30 Read block 171 len 8192 19/30 Write block 546 len 28672 data: 7 6 6 5 5 7 8 20/30 Read block 371 len 32768 21/30 Read block 60 len 36864 22/30 Write block 55 len 8192 data: 8 11 23/30 Write block 109 len 28672 data: 9 10 10 10 9 9 8 24/30 Write block 136 len 36864 data: 7 8 10 11 11 9 8 7 7 25/30 Write block 393 len 28672 data: 9 8 5 8 9 8 6 26/30 Write block 110 len 16384 data: 11 11 11 10 27/30 Write block 367 len 36864 data: 4 3 4 5 8 8 7 8 11 28/30 Write block 491 len 40960 data: 9 8 7 6 6 5 5 7 7 8 29/30 Read block 357 len 32768 30/30 Write block 450 len 32768 data: 5 6 5 5 6 5 8 6 ---------------------------------------------------------------- Crucible gen:82 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 8 Done Done Done false 1002 Acked Read 1 Done Sent Done false 1003 Acked Read 7 Done Sent Done false 1004 Acked Read 5 Done Sent Done false 1005 Acked Read 1 Done Sent Done false 1006 Acked Write 9 Done Sent Done false 1007 Acked Read 5 Done Sent Done false 1008 Acked Read 9 Done Sent Done false 1009 Acked Read 10 Done Sent Done false 1010 Acked Read 10 Done Sent Done false 1011 Acked Read 5 Done Sent Done false 1012 Acked Write 7 Done Sent Done false 1013 Acked Read 5 Done Sent Done false 1014 Acked Read 4 Done Sent Done false 1015 Acked Read 7 Done Sent Done false 1016 Acked Read 8 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Read 8 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Write 2 Done Sent Done false 1022 Acked Write 7 Done Sent Done false 1023 Acked Write 9 Done Sent Done false 1024 Acked Write 7 Done Sent Done false 1025 Acked Write 4 Done Sent Done false 1026 Acked Write 9 Done Sent Done false 1027 Acked Write 10 Done Sent Done false 1028 Acked Read 8 Done Sent Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 28 1 30 Done 29 2 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:12.904571452Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:12.904753038Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1461,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 78 78 78 145 145 145 T T T 1 020-039 78 78 78 145 145 145 F F F 2 040-059 81 81 81 149 149 149 T F T <--- 3 060-079 81 81 81 149 149 149 F F F 4 080-099 78 78 78 144 144 144 T T T 5 100-119 80 80 80 148 148 148 T F T <--- 6 120-139 81 81 81 149 149 149 T F T <--- 7 140-159 81 81 81 149 149 149 T F T <--- 8 160-179 76 76 76 142 142 142 F F F 9 180-199 81 81 81 149 149 149 F F F 10 200-219 81 81 81 149 149 149 F F F 11 220-239 78 78 78 144 144 144 F F F 12 240-259 80 80 80 148 148 148 F F F 13 260-279 80 80 80 148 148 148 F F F 14 280-299 77 77 77 143 143 143 F F F 15 300-319 81 81 81 149 149 149 F F F 16 320-339 76 76 76 142 142 142 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 76 76 76 140 140 140 T F T <--- 19 380-399 78 78 78 146 146 146 T F T <--- 20 400-419 81 81 81 149 149 149 F F F 21 420-439 77 77 77 143 143 143 F F F 22 440-459 79 79 79 147 147 147 T F T <--- 23 460-479 74 74 74 135 135 135 F F F 24 480-499 72 72 72 129 129 129 T F T <--- 25 500-519 78 78 78 145 145 145 T F T <--- 26 520-539 81 81 81 149 149 149 F F F 27 540-559 81 81 81 149 149 149 T F T <--- 28 560-579 76 76 76 140 140 140 F F F 29 580-599 76 76 76 142 142 142 T F T <--- Max gen: 81, Max flush: 149 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 41 Verifying data now No change after verify Loop: 41 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 82 82 82 150 150 150 F F F 1 020-039 78 78 78 145 145 145 F F F 2 040-059 82 82 82 150 150 150 F F F 3 060-079 81 81 81 149 149 149 F F F 4 080-099 82 82 82 150 150 150 F F F 5 100-119 82 82 82 150 150 150 F F F 6 120-139 82 82 82 150 150 150 F F F 7 140-159 82 82 82 150 150 150 F F F 8 160-179 76 76 76 142 142 142 F F F 9 180-199 81 81 81 149 149 149 F F F 10 200-219 81 81 81 149 149 149 F F F 11 220-239 78 78 78 144 144 144 F F F 12 240-259 80 80 80 148 148 148 F F F 13 260-279 80 80 80 148 148 148 F F F 14 280-299 77 77 77 143 143 143 F F F 15 300-319 81 81 81 149 149 149 F F F 16 320-339 76 76 76 142 142 142 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 82 82 82 150 150 150 F F F 19 380-399 82 82 82 150 150 150 F F F 20 400-419 81 81 81 149 149 149 F F F 21 420-439 77 77 77 143 143 143 F F F 22 440-459 82 82 82 150 150 150 F F F 23 460-479 74 74 74 135 135 135 F F F 24 480-499 82 82 82 150 150 150 F F F 25 500-519 82 82 82 150 150 150 F F F 26 520-539 81 81 81 149 149 149 F F F 27 540-559 82 82 82 150 150 150 F F F 28 560-579 76 76 76 140 140 140 F F F 29 580-599 82 82 82 150 150 150 F F F Max gen: 82, Max flush: 150 Begin loop 42 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.159489023Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:17.166664719Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.166716499Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.173590772Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.173863711Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.173899394Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.173908272Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.17391973Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469} {"msg":"Crucible 0d4c9a4a-36e3-4713-9626-7b5ddde33473 has session id: c8f409c2-10c8-4aa2-a784-a66290ebf83e","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174256607Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"Upstairs opts: Upstairs UUID: 0d4c9a4a-36e3-4713-9626-7b5ddde33473, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174280773Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"Crucible stats registered with UUID: 0d4c9a4a-36e3-4713-9626-7b5ddde33473","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174290531Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:17.174301029Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174448751Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"io task","client":"0","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174477596Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"io task","client":"1","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174487954Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"io task","client":"2","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174630947Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"task":"crutest"} {"msg":"The guest has requested activation with gen:84","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174655102Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469} {"msg":"0d4c9a4a-36e3-4713-9626-7b5ddde33473 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174672879Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174683267Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"client":"0","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174693065Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"client":"1","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174702683Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"client":"2","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174712232Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"io task","client":"2","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.17472222Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"io task","client":"0","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174732308Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"io task","client":"1","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"ds_connection connected from Ok(127.0.0.1:49761)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174849156Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"io task","client":"0","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"ds_connection connected from Ok(127.0.0.1:38960)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174909705Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"io task","client":"1","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"ds_connection connected from Ok(127.0.0.1:34129)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.174921523Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"io task","client":"2","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175450913Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"client":"1","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175463401Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"client":"2","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175569631Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"client":"0","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175742399Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175778022Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"[0]R flush_numbers[0..12]: [150, 145, 150, 149, 150, 150, 150, 150, 142, 149, 149, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175830272Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"downstairs","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"[0]R generation[0..12]: [82, 78, 82, 81, 82, 82, 82, 82, 76, 81, 81, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.17584446Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"downstairs","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175855848Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"downstairs","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"[1]R flush_numbers[0..12]: [150, 145, 150, 149, 150, 150, 150, 150, 142, 149, 149, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175866576Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"downstairs","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"[1]R generation[0..12]: [82, 78, 82, 81, 82, 82, 82, 82, 76, 81, 81, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175876924Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"downstairs","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175887942Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"downstairs","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"[2]R flush_numbers[0..12]: [150, 145, 150, 149, 150, 150, 150, 150, 142, 149, 149, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.1758985Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"downstairs","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"[2]R generation[0..12]: [82, 78, 82, 81, 82, 82, 82, 82, 76, 81, 81, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175909178Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"downstairs","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175920026Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"downstairs","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"Max found gen is 83","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175929404Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"downstairs","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"Generation requested: 84 >= found:83","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175939142Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"downstairs","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"Next flush: 151","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.17594844Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"downstairs","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175958078Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"":"downstairs","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175967777Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175976425Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"client":"0","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.175987533Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"client":"1","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.176012878Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"client":"2","":"downstairs","session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.176022736Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.176031725Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"0d4c9a4a-36e3-4713-9626-7b5ddde33473 is now active with session: c8f409c2-10c8-4aa2-a784-a66290ebf83e","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.176041553Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.176050191Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"session_id":"c8f409c2-10c8-4aa2-a784-a66290ebf83e"} {"msg":"The guest has finished waiting for activation with:84","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.176059199Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469} {"msg":"Activated sub_volume 0d4c9a4a-36e3-4713-9626-7b5ddde33473","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.176067368Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.176075586Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.176388078Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"task":"crutest"} 01/30 Flush 02/30 Write block 33 len 36864 data: 8 7 6 5 6 7 7 7 8 03/30 Write block 453 len 32768 data: 6 7 6 9 7 4 4 4 04/30 Write block 282 len 8192 data: 3 3 05/30 Read block 77 len 4096 06/30 Read block 152 len 40960 07/30 Flush 08/30 Write block 138 len 8192 data: 11 12 09/30 Write block 552 len 4096 data: 9 10/30 Write block 473 len 16384 data: 5 4 5 4 11/30 Write block 30 len 28672 data: 10 10 11 9 8 7 6 12/30 Read block 78 len 36864 13/30 Write block 543 len 12288 data: 8 8 8 14/30 Read block 50 len 40960 15/30 Write block 592 len 8192 data: 11 13 16/30 Read block 141 len 16384 17/30 Write block 192 len 28672 data: 11 13 10 11 11 7 6 18/30 Read block 86 len 20480 19/30 Read block 529 len 8192 20/30 Read block 435 len 32768 21/30 Write block 302 len 20480 data: 10 12 13 10 10 22/30 Write block 139 len 36864 data: 13 12 10 9 8 8 7 5 6 23/30 Read block 438 len 28672 24/30 Flush 25/30 Write block 258 len 40960 data: 9 8 8 8 8 7 5 3 4 7 26/30 Write block 459 len 4096 data: 5 27/30 Write block 471 len 8192 data: 8 6 28/30 Read block 179 len 36864 29/30 Read block 406 len 20480 30/30 Read block 493 len 4096 ---------------------------------------------------------------- Crucible gen:84 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Sent Done Done false 1001 Acked Write 9 Sent Done Done false 1002 Acked Write 8 Sent Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Read 10 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Write 1 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Write 7 Sent Done Done false 1011 Acked Read 9 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Read 10 Sent Done Done false 1014 Acked Write 2 Sent Done Done false 1015 Acked Read 4 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Read 5 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Read 8 Sent Done Done false 1020 Acked Write 5 Sent Done Done false 1021 Acked Write 9 Sent Done Done false 1022 Acked Read 7 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 10 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Write 2 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Read 1 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 0 0 30 Done 0 30 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:17.185157975Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:17.185334002Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1469,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 82 82 82 150 150 150 F F F 1 020-039 78 84 84 145 153 153 F F F <--- 2 040-059 82 84 84 150 152 152 F F F <--- 3 060-079 81 81 81 149 149 149 F F F 4 080-099 82 82 82 150 150 150 F F F 5 100-119 82 82 82 150 150 150 F F F 6 120-139 82 84 84 150 153 153 F F F <--- 7 140-159 82 84 84 150 153 153 F F F <--- 8 160-179 76 76 76 142 142 142 F F F 9 180-199 81 84 84 149 153 153 F F F <--- 10 200-219 81 81 81 149 149 149 F F F 11 220-239 78 78 78 144 144 144 F F F 12 240-259 80 80 80 148 148 148 F T T <--- 13 260-279 80 80 80 148 148 148 F T T <--- 14 280-299 77 84 84 143 152 152 F F F <--- 15 300-319 81 84 84 149 153 153 F F F <--- 16 320-339 76 76 76 142 142 142 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 82 82 82 150 150 150 F F F 19 380-399 82 82 82 150 150 150 F F F 20 400-419 81 81 81 149 149 149 F F F 21 420-439 77 77 77 143 143 143 F F F 22 440-459 82 84 84 150 152 152 F T T <--- 23 460-479 74 84 84 135 153 153 F T T <--- 24 480-499 82 82 82 150 150 150 F F F 25 500-519 82 82 82 150 150 150 F F F 26 520-539 81 81 81 149 149 149 F F F 27 540-559 82 84 84 150 153 153 F F F <--- 28 560-579 76 76 76 140 140 140 F F F 29 580-599 82 84 84 150 153 153 F F F <--- Max gen: 84, Max flush: 153 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 42 Verifying data now No change after verify Loop: 42 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 82 82 82 150 150 150 F F F 1 020-039 84 84 84 153 153 153 F F F 2 040-059 84 84 84 152 152 152 F F F 3 060-079 81 81 81 149 149 149 F F F 4 080-099 82 82 82 150 150 150 F F F 5 100-119 82 82 82 150 150 150 F F F 6 120-139 84 84 84 153 153 153 F F F 7 140-159 84 84 84 153 153 153 F F F 8 160-179 76 76 76 142 142 142 F F F 9 180-199 84 84 84 153 153 153 F F F 10 200-219 81 81 81 149 149 149 F F F 11 220-239 78 78 78 144 144 144 F F F 12 240-259 85 85 85 154 154 154 F F F 13 260-279 85 85 85 154 154 154 F F F 14 280-299 84 84 84 152 152 152 F F F 15 300-319 84 84 84 153 153 153 F F F 16 320-339 76 76 76 142 142 142 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 82 82 82 150 150 150 F F F 19 380-399 82 82 82 150 150 150 F F F 20 400-419 81 81 81 149 149 149 F F F 21 420-439 77 77 77 143 143 143 F F F 22 440-459 85 85 85 154 154 154 F F F 23 460-479 85 85 85 154 154 154 F F F 24 480-499 82 82 82 150 150 150 F F F 25 500-519 82 82 82 150 150 150 F F F 26 520-539 81 81 81 149 149 149 F F F 27 540-559 84 84 84 153 153 153 F F F 28 560-579 76 76 76 140 140 140 F F F 29 580-599 84 84 84 153 153 153 F F F Max gen: 85, Max flush: 154 Begin loop 43 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.439804587Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:21.446947149Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.447002069Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.453752455Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454026773Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454053118Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454064916Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454076494Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477} {"msg":"Crucible ce12ed0d-b187-483b-88b7-62d6837d558c has session id: be4aefca-d1c8-4fb0-83f1-a032b35c3d87","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454413811Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"Upstairs opts: Upstairs UUID: ce12ed0d-b187-483b-88b7-62d6837d558c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454442375Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"Crucible stats registered with UUID: ce12ed0d-b187-483b-88b7-62d6837d558c","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454453593Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:21.454462852Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454608284Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"io task","client":"1","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454625571Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"io task","client":"0","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454635509Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"io task","client":"2","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454764525Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"task":"crutest"} {"msg":"The guest has requested activation with gen:86","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454777103Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477} {"msg":"ce12ed0d-b187-483b-88b7-62d6837d558c active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454795119Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454803828Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"client":"0","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454813396Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"client":"1","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454822974Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"client":"2","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454833722Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"io task","client":"2","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.4548437Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"io task","client":"0","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454854078Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"io task","client":"1","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"ds_connection connected from Ok(127.0.0.1:65426)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454915237Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"io task","client":"2","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"ds_connection connected from Ok(127.0.0.1:33181)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454926085Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"io task","client":"1","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"ds_connection connected from Ok(127.0.0.1:43132)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.454936343Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"io task","client":"0","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.455638051Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"client":"2","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.455663267Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"client":"1","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.455675075Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"client":"0","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.455945724Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.455955572Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"[0]R flush_numbers[0..12]: [150, 153, 152, 149, 150, 150, 153, 153, 142, 153, 149, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456035087Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"downstairs","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"[0]R generation[0..12]: [82, 84, 84, 81, 82, 82, 84, 84, 76, 84, 81, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456046475Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"downstairs","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456057543Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"downstairs","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"[1]R flush_numbers[0..12]: [150, 153, 152, 149, 150, 150, 153, 153, 142, 153, 149, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.45607073Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"downstairs","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"[1]R generation[0..12]: [82, 84, 84, 81, 82, 82, 84, 84, 76, 84, 81, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456081418Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"downstairs","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456092196Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"downstairs","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"[2]R flush_numbers[0..12]: [150, 153, 152, 149, 150, 150, 153, 153, 142, 153, 149, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456102714Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"downstairs","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"[2]R generation[0..12]: [82, 84, 84, 81, 82, 82, 84, 84, 76, 84, 81, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456112973Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"downstairs","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.45612383Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"downstairs","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"Max found gen is 86","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456133209Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"downstairs","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"Generation requested: 86 >= found:86","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456142897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"downstairs","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"Next flush: 155","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456152355Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"downstairs","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456161703Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"":"downstairs","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456171442Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.45618004Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"client":"0","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456189758Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"client":"1","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456199626Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"client":"2","":"downstairs","session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456209674Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456220172Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"ce12ed0d-b187-483b-88b7-62d6837d558c is now active with session: be4aefca-d1c8-4fb0-83f1-a032b35c3d87","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456230081Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456238769Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"session_id":"be4aefca-d1c8-4fb0-83f1-a032b35c3d87"} {"msg":"The guest has finished waiting for activation with:86","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456247717Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477} {"msg":"Activated sub_volume ce12ed0d-b187-483b-88b7-62d6837d558c","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456255736Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456263754Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.456525265Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"task":"crutest"} 01/30 Write block 395 len 40960 data: 6 9 10 9 7 4 5 7 7 8 02/30 Write block 567 len 12288 data: 5 5 5 03/30 Write block 191 len 40960 data: 10 12 14 11 12 12 8 7 5 4 04/30 Write block 103 len 32768 data: 7 7 7 7 10 8 10 12 05/30 Read block 60 len 40960 06/30 Read block 451 len 36864 07/30 Read block 33 len 28672 08/30 Write block 21 len 24576 data: 7 7 8 7 7 7 09/30 Read block 340 len 4096 10/30 Read block 90 len 16384 11/30 Write block 161 len 28672 data: 4 5 5 5 6 7 6 12/30 Read block 119 len 20480 13/30 Write block 37 len 8192 data: 7 8 14/30 Read block 217 len 36864 15/30 Write block 133 len 36864 data: 12 10 9 8 9 12 14 13 11 16/30 Read block 485 len 12288 17/30 Write block 315 len 12288 data: 8 9 10 18/30 Read block 391 len 28672 19/30 Read block 40 len 12288 20/30 Flush 21/30 Write block 388 len 28672 data: 7 5 7 8 9 10 9 22/30 Write block 406 len 4096 data: 6 23/30 Write block 403 len 16384 data: 8 9 6 7 24/30 Write block 141 len 32768 data: 12 10 9 9 8 6 7 4 25/30 Read block 501 len 40960 26/30 Write block 414 len 12288 data: 5 5 5 27/30 Read block 131 len 40960 28/30 Read block 53 len 36864 29/30 Read block 531 len 32768 30/30 Read block 580 len 28672 ---------------------------------------------------------------- Crucible gen:86 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Sent Done Done false 1001 Acked Write 3 Sent Done Done false 1002 Acked Write 10 Sent Done Done false 1003 Acked Write 8 Sent Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Read 9 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Read 1 Sent Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Write 7 Sent Done Done false 1011 Acked Read 5 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Read 9 Sent Done Done false 1014 Acked Write 9 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Write 3 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Read 3 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Write 1 Sent Done Done false 1022 Acked Write 4 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Read 8 Sent Done Done false 1029 Acked Read 7 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 0 31 Done 0 29 30 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:21.465773813Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:21.46594859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1477,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 82 82 82 150 150 150 F F F 1 020-039 84 86 86 153 155 155 F F F <--- 2 040-059 84 84 84 152 152 152 F F F 3 060-079 81 81 81 149 149 149 F F F 4 080-099 82 82 82 150 150 150 F F F 5 100-119 82 86 86 150 155 155 F F F <--- 6 120-139 84 86 86 153 155 155 F F F <--- 7 140-159 84 86 86 153 155 155 F T T <--- 8 160-179 76 86 86 142 155 155 F F F <--- 9 180-199 84 86 86 153 155 155 F F F <--- 10 200-219 81 86 86 149 155 155 F F F <--- 11 220-239 78 78 78 144 144 144 F F F 12 240-259 85 85 85 154 154 154 F F F 13 260-279 85 85 85 154 154 154 F F F 14 280-299 84 84 84 152 152 152 F F F 15 300-319 84 86 86 153 155 155 F F F <--- 16 320-339 76 76 76 142 142 142 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 82 82 82 150 150 150 F F F 19 380-399 82 86 86 150 155 155 F T T <--- 20 400-419 81 86 86 149 155 155 F T T <--- 21 420-439 77 77 77 143 143 143 F F F 22 440-459 85 85 85 154 154 154 F F F 23 460-479 85 85 85 154 154 154 F F F 24 480-499 82 82 82 150 150 150 F F F 25 500-519 82 82 82 150 150 150 F F F 26 520-539 81 81 81 149 149 149 F F F 27 540-559 84 84 84 153 153 153 F F F 28 560-579 76 86 86 140 155 155 F F F <--- 29 580-599 84 84 84 153 153 153 F F F Max gen: 86, Max flush: 155 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 43 Verifying data now No change after verify Loop: 43 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 82 82 82 150 150 150 F F F 1 020-039 86 86 86 155 155 155 F F F 2 040-059 84 84 84 152 152 152 F F F 3 060-079 81 81 81 149 149 149 F F F 4 080-099 82 82 82 150 150 150 F F F 5 100-119 86 86 86 155 155 155 F F F 6 120-139 86 86 86 155 155 155 F F F 7 140-159 87 87 87 156 156 156 F F F 8 160-179 86 86 86 155 155 155 F F F 9 180-199 86 86 86 155 155 155 F F F 10 200-219 86 86 86 155 155 155 F F F 11 220-239 78 78 78 144 144 144 F F F 12 240-259 85 85 85 154 154 154 F F F 13 260-279 85 85 85 154 154 154 F F F 14 280-299 84 84 84 152 152 152 F F F 15 300-319 86 86 86 155 155 155 F F F 16 320-339 76 76 76 142 142 142 F F F 17 340-359 70 70 70 126 126 126 F F F 18 360-379 82 82 82 150 150 150 F F F 19 380-399 87 87 87 156 156 156 F F F 20 400-419 87 87 87 156 156 156 F F F 21 420-439 77 77 77 143 143 143 F F F 22 440-459 85 85 85 154 154 154 F F F 23 460-479 85 85 85 154 154 154 F F F 24 480-499 82 82 82 150 150 150 F F F 25 500-519 82 82 82 150 150 150 F F F 26 520-539 81 81 81 149 149 149 F F F 27 540-559 84 84 84 153 153 153 F F F 28 560-579 86 86 86 155 155 155 F F F 29 580-599 84 84 84 153 153 153 F F F Max gen: 87, Max flush: 156 Begin loop 44 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.719715137Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.727905013Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.728201867Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.728230112Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.72823874Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.728250298Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485} {"msg":"Crucible 8f2f5229-6524-4688-b3e0-68c8ee4652fc has session id: 2bd4ba54-b942-47b7-b4d1-40798e9d03ea","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.728580996Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"Upstairs opts: Upstairs UUID: 8f2f5229-6524-4688-b3e0-68c8ee4652fc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.728611771Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"Crucible stats registered with UUID: 8f2f5229-6524-4688-b3e0-68c8ee4652fc","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.728624338Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:25.728633257Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.728779489Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"io task","client":"1","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.728810593Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"io task","client":"0","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.728821611Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"io task","client":"2","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.728951117Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"task":"crutest"} {"msg":"The guest has requested activation with gen:88","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.728968084Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485} {"msg":"8f2f5229-6524-4688-b3e0-68c8ee4652fc active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.729003667Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.729013275Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"client":"0","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.729022924Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"client":"1","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.729032612Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"client":"2","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.72904203Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"io task","client":"2","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.729052148Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"io task","client":"0","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.729062116Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"io task","client":"1","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"ds_connection connected from Ok(127.0.0.1:42632)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.729114876Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"io task","client":"1","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"ds_connection connected from Ok(127.0.0.1:45794)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.729141591Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"io task","client":"0","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"ds_connection connected from Ok(127.0.0.1:64543)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.729159938Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"io task","client":"2","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.729679231Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"client":"0","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.729748488Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"client":"2","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.72978833Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"client":"1","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.729975315Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730043252Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"[0]R flush_numbers[0..12]: [150, 155, 152, 149, 150, 155, 155, 156, 155, 155, 155, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730080865Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"downstairs","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"[0]R generation[0..12]: [82, 86, 84, 81, 82, 86, 86, 87, 86, 86, 86, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730092053Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"downstairs","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730103131Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"downstairs","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"[1]R flush_numbers[0..12]: [150, 155, 152, 149, 150, 155, 155, 156, 155, 155, 155, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730113669Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"downstairs","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"[1]R generation[0..12]: [82, 86, 84, 81, 82, 86, 86, 87, 86, 86, 86, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730124057Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"downstairs","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730135035Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"downstairs","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"[2]R flush_numbers[0..12]: [150, 155, 152, 149, 150, 155, 155, 156, 155, 155, 155, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730145683Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"downstairs","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"[2]R generation[0..12]: [82, 86, 84, 81, 82, 86, 86, 87, 86, 86, 86, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730159091Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"downstairs","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730170019Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"downstairs","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"Max found gen is 88","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730179537Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"downstairs","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"Generation requested: 88 >= found:88","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730189215Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"downstairs","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"Next flush: 157","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730198603Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"downstairs","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730207962Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"":"downstairs","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.73021814Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730226958Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"client":"0","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730236506Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"client":"1","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730245984Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"client":"2","":"downstairs","session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730255563Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730264481Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"8f2f5229-6524-4688-b3e0-68c8ee4652fc is now active with session: 2bd4ba54-b942-47b7-b4d1-40798e9d03ea","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730274199Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730282738Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"session_id":"2bd4ba54-b942-47b7-b4d1-40798e9d03ea"} {"msg":"The guest has finished waiting for activation with:88","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730291696Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485} {"msg":"Activated sub_volume 8f2f5229-6524-4688-b3e0-68c8ee4652fc","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730299644Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730307613Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.730604777Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"task":"crutest"} 01/30 Write block 57 len 16384 data: 10 9 11 11 02/30 Read block 140 len 12288 03/30 Write block 510 len 32768 data: 5 5 5 5 6 7 7 7 04/30 Read block 110 len 4096 05/30 Read block 372 len 12288 06/30 Read block 531 len 32768 07/30 Write block 288 len 28672 data: 7 8 8 6 7 7 5 08/30 Write block 351 len 40960 data: 7 6 6 6 5 7 6 3 3 3 09/30 Write block 575 len 32768 data: 6 6 6 8 7 6 7 7 10/30 Write block 177 len 20480 data: 7 11 9 7 6 11/30 Read block 558 len 8192 12/30 Read block 183 len 12288 13/30 Write block 518 len 24576 data: 6 6 6 8 8 8 14/30 Write block 134 len 36864 data: 11 10 9 10 13 15 14 13 11 15/30 Read block 563 len 24576 16/30 Read block 519 len 16384 17/30 Write block 424 len 16384 data: 5 5 5 6 18/30 Read block 517 len 20480 19/30 Read block 352 len 4096 20/30 Read block 375 len 24576 21/30 Write block 549 len 24576 data: 6 6 8 10 9 9 22/30 Read block 478 len 20480 23/30 Read block 115 len 40960 24/30 Write block 577 len 40960 data: 7 9 8 7 8 8 6 5 4 6 25/30 Read block 314 len 8192 26/30 Write block 210 len 28672 data: 11 9 12 11 13 13 12 27/30 Read block 457 len 16384 28/30 Write block 130 len 28672 data: 14 12 12 13 12 11 10 29/30 Write block 108 len 12288 data: 9 11 13 30/30 Write block 345 len 20480 data: 7 6 4 3 6 ---------------------------------------------------------------- Crucible gen:88 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 3 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Read 1 Done Sent Done false 1004 Acked Read 3 Done Sent Done false 1005 Acked Read 8 Done Sent Done false 1006 Acked Write 7 Done Sent Done false 1007 Acked Write 10 Done Sent Done false 1008 Acked Write 8 Done Sent Done false 1009 Acked Write 5 Done Sent Done false 1010 Acked Read 2 Done Sent Done false 1011 Acked Read 3 Done Sent Done false 1012 Acked Write 6 Done Sent Done false 1013 Acked Write 9 Done Sent Done false 1014 Acked Read 6 Done Sent Done false 1015 Acked Read 4 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Read 1 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Write 6 Done Sent Done false 1021 Acked Read 5 Done Sent Done false 1022 Acked Read 10 Done Sent Done false 1023 Acked Write 10 Done Sent Done false 1024 Acked Read 2 Done Sent Done false 1025 Acked Write 7 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Write 7 Sent Sent Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 27 3 33 Done 27 3 27 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:25.739628077Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:25.739812343Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1485,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 82 82 82 150 150 150 F F F 1 020-039 86 86 86 155 155 155 F F F 2 040-059 84 84 84 152 152 152 T T T 3 060-079 81 81 81 149 149 149 T T T 4 080-099 82 82 82 150 150 150 F F F 5 100-119 86 86 86 155 155 155 T F T <--- 6 120-139 86 86 86 155 155 155 T F T <--- 7 140-159 87 87 87 156 156 156 T F T <--- 8 160-179 86 86 86 155 155 155 T F T <--- 9 180-199 86 86 86 155 155 155 T F T <--- 10 200-219 86 86 86 155 155 155 T F T <--- 11 220-239 78 78 78 144 144 144 F F F 12 240-259 85 85 85 154 154 154 F F F 13 260-279 85 85 85 154 154 154 F F F 14 280-299 84 84 84 152 152 152 T F T <--- 15 300-319 86 86 86 155 155 155 F F F 16 320-339 76 76 76 142 142 142 F F F 17 340-359 70 70 70 126 126 126 T F T <--- 18 360-379 82 82 82 150 150 150 T F T <--- 19 380-399 87 87 87 156 156 156 F F F 20 400-419 87 87 87 156 156 156 F F F 21 420-439 77 77 77 143 143 143 T F T <--- 22 440-459 85 85 85 154 154 154 F F F 23 460-479 85 85 85 154 154 154 F F F 24 480-499 82 82 82 150 150 150 F F F 25 500-519 82 82 82 150 150 150 T T T 26 520-539 81 81 81 149 149 149 T F T <--- 27 540-559 84 84 84 153 153 153 T F T <--- 28 560-579 86 86 86 155 155 155 T F T <--- 29 580-599 84 84 84 153 153 153 T F T <--- Max gen: 87, Max flush: 156 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 44 Verifying data now No change after verify Loop: 44 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 82 82 82 150 150 150 F F F 1 020-039 86 86 86 155 155 155 F F F 2 040-059 88 88 88 157 157 157 F F F 3 060-079 88 88 88 157 157 157 F F F 4 080-099 82 82 82 150 150 150 F F F 5 100-119 88 88 88 157 157 157 F F F 6 120-139 88 88 88 157 157 157 F F F 7 140-159 88 88 88 157 157 157 F F F 8 160-179 88 88 88 157 157 157 F F F 9 180-199 88 88 88 157 157 157 F F F 10 200-219 88 88 88 157 157 157 F F F 11 220-239 78 78 78 144 144 144 F F F 12 240-259 85 85 85 154 154 154 F F F 13 260-279 85 85 85 154 154 154 F F F 14 280-299 88 88 88 157 157 157 F F F 15 300-319 86 86 86 155 155 155 F F F 16 320-339 76 76 76 142 142 142 F F F 17 340-359 88 88 88 157 157 157 F F F 18 360-379 88 88 88 157 157 157 F F F 19 380-399 87 87 87 156 156 156 F F F 20 400-419 87 87 87 156 156 156 F F F 21 420-439 88 88 88 157 157 157 F F F 22 440-459 85 85 85 154 154 154 F F F 23 460-479 85 85 85 154 154 154 F F F 24 480-499 82 82 82 150 150 150 F F F 25 500-519 88 88 88 157 157 157 F F F 26 520-539 88 88 88 157 157 157 F F F 27 540-559 88 88 88 157 157 157 F F F 28 560-579 88 88 88 157 157 157 F F F 29 580-599 88 88 88 157 157 157 F F F Max gen: 88, Max flush: 157 Begin loop 45 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.005102141Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.013300845Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.013602549Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.013631463Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.013639892Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.013651159Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493} {"msg":"Crucible 55591038-a80b-47ff-8bb1-8715277d0296 has session id: 8c273265-3ac1-475a-a14e-a5f4d994fe98","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014041806Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"Upstairs opts: Upstairs UUID: 55591038-a80b-47ff-8bb1-8715277d0296, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014070701Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"Crucible stats registered with UUID: 55591038-a80b-47ff-8bb1-8715277d0296","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014081979Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:30.014091387Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014216404Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"io task","client":"1","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.01423764Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"io task","client":"0","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014248068Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"io task","client":"2","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014402809Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"task":"crutest"} {"msg":"The guest has requested activation with gen:90","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014426764Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493} {"msg":"55591038-a80b-47ff-8bb1-8715277d0296 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014443001Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.0144518Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"client":"0","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014461858Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"client":"1","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014471606Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"client":"2","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014481054Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"io task","client":"2","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014490802Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"io task","client":"0","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014508149Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"io task","client":"1","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"ds_connection connected from Ok(127.0.0.1:42812)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.014583545Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"io task","client":"2","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"ds_connection connected from Ok(127.0.0.1:36245)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.0146092Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"io task","client":"1","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"ds_connection connected from Ok(127.0.0.1:42746)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.01471715Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"io task","client":"0","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015308549Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"client":"1","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015321587Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"client":"2","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015335244Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"client":"0","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015619601Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015652275Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"[0]R flush_numbers[0..12]: [150, 155, 157, 157, 150, 157, 157, 157, 157, 157, 157, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015723541Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"downstairs","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"[0]R generation[0..12]: [82, 86, 88, 88, 82, 88, 88, 88, 88, 88, 88, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015734509Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"downstairs","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015745557Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"downstairs","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"[1]R flush_numbers[0..12]: [150, 155, 157, 157, 150, 157, 157, 157, 157, 157, 157, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015756105Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"downstairs","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"[1]R generation[0..12]: [82, 86, 88, 88, 82, 88, 88, 88, 88, 88, 88, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015766333Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"downstairs","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015776951Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"downstairs","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"[2]R flush_numbers[0..12]: [150, 155, 157, 157, 150, 157, 157, 157, 157, 157, 157, 144]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015787459Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"downstairs","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"[2]R generation[0..12]: [82, 86, 88, 88, 82, 88, 88, 88, 88, 88, 88, 78]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015797627Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"downstairs","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015808365Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"downstairs","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"Max found gen is 89","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015817894Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"downstairs","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"Generation requested: 90 >= found:89","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015827542Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"downstairs","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"Next flush: 158","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.01583831Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"downstairs","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015847648Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"":"downstairs","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015857286Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015866055Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"client":"0","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015875683Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"client":"1","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015885401Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"client":"2","":"downstairs","session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015895139Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015904148Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"55591038-a80b-47ff-8bb1-8715277d0296 is now active with session: 8c273265-3ac1-475a-a14e-a5f4d994fe98","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015913906Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015922674Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"session_id":"8c273265-3ac1-475a-a14e-a5f4d994fe98"} {"msg":"The guest has finished waiting for activation with:90","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015931562Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493} {"msg":"Activated sub_volume 55591038-a80b-47ff-8bb1-8715277d0296","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015939821Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.015947919Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.016240744Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"task":"crutest"} 01/30 Write block 229 len 40960 data: 5 6 5 6 5 5 4 5 6 6 02/30 Read block 244 len 32768 03/30 Write block 256 len 16384 data: 8 8 10 9 04/30 Write block 199 len 36864 data: 6 5 4 6 5 5 4 6 7 05/30 Write block 379 len 32768 data: 9 11 9 9 10 9 7 8 06/30 Write block 232 len 28672 data: 7 6 6 5 6 7 7 07/30 Read block 218 len 8192 08/30 Flush 09/30 Write block 1 len 28672 data: 2 2 2 2 2 5 6 10/30 Write block 90 len 12288 data: 5 6 4 11/30 Write block 28 len 24576 data: 7 9 11 11 12 10 12/30 Write block 265 len 40960 data: 4 5 8 6 7 6 5 3 3 6 13/30 Read block 281 len 20480 14/30 Read block 528 len 12288 15/30 Write block 234 len 12288 data: 7 6 7 16/30 Read block 583 len 8192 17/30 Read block 296 len 12288 18/30 Flush 19/30 Write block 419 len 12288 data: 4 3 3 20/30 Read block 300 len 16384 21/30 Read block 49 len 20480 22/30 Read block 105 len 28672 23/30 Read block 591 len 12288 24/30 Write block 129 len 24576 data: 12 15 13 13 14 13 25/30 Read block 209 len 4096 26/30 Read block 478 len 32768 27/30 Read block 571 len 32768 28/30 Read block 507 len 24576 29/30 Read block 312 len 4096 30/30 Write block 444 len 36864 data: 6 7 5 4 2 4 6 7 6 ---------------------------------------------------------------- Crucible gen:90 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Sent Done false 1001 Acked Read 8 Done Sent Done false 1002 Acked Write 4 Done Sent Done false 1003 Acked Write 9 Done Sent Done false 1004 Acked Write 8 Done Sent Done false 1005 Acked Write 7 Done Sent Done false 1006 Acked Read 2 Done Sent Done false 1007 Acked Flush 0 Done Sent Done false 1008 Acked Write 7 Done Sent Done false 1009 Acked Write 3 Done Sent Done false 1010 Acked Write 6 Done Sent Done false 1011 Acked Write 10 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Read 3 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Read 2 Done Sent Done false 1016 Acked Read 3 Done Sent Done false 1017 Acked Flush 0 Done Sent Done false 1018 Acked Write 3 Done Sent Done false 1019 Acked Read 4 Done Sent Done false 1020 Acked Read 5 Done Sent Done false 1021 Acked Read 7 Done Sent Done false 1022 Acked Read 3 Done Sent Done false 1023 Acked Write 6 Done Sent Done false 1024 Acked Read 1 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Read 1 Done Sent Done false 1029 Acked Write 9 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 1 32 Done 29 0 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1017)) None Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:30.024954303Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:30.025159334Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1493,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 82 90 159 150 159 F F F <--- 1 020-039 90 86 90 159 155 159 F F F <--- 2 040-059 88 88 88 157 157 157 F F F 3 060-079 88 88 88 157 157 157 F F F 4 080-099 90 82 90 159 150 159 F F F <--- 5 100-119 88 88 88 157 157 157 F F F 6 120-139 88 88 88 157 157 157 T F T <--- 7 140-159 88 88 88 157 157 157 F F F 8 160-179 88 88 88 157 157 157 F F F 9 180-199 90 88 90 158 157 158 F F F <--- 10 200-219 90 88 90 158 157 158 F F F <--- 11 220-239 90 78 90 159 144 159 F F F <--- 12 240-259 90 85 90 158 154 158 F F F <--- 13 260-279 90 85 90 159 154 159 F F F <--- 14 280-299 88 88 88 157 157 157 F F F 15 300-319 86 86 86 155 155 155 F F F 16 320-339 76 76 76 142 142 142 F F F 17 340-359 88 88 88 157 157 157 F F F 18 360-379 90 88 90 158 157 158 F F F <--- 19 380-399 90 87 90 158 156 158 F F F <--- 20 400-419 87 87 87 156 156 156 T F T <--- 21 420-439 88 88 88 157 157 157 T F T <--- 22 440-459 85 85 85 154 154 154 T F T <--- 23 460-479 85 85 85 154 154 154 F F F 24 480-499 82 82 82 150 150 150 F F F 25 500-519 88 88 88 157 157 157 F F F 26 520-539 88 88 88 157 157 157 F F F 27 540-559 88 88 88 157 157 157 F F F 28 560-579 88 88 88 157 157 157 F F F 29 580-599 88 88 88 157 157 157 F F F Max gen: 90, Max flush: 159 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 45 Verifying data now No change after verify Loop: 45 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 159 159 159 F F F 1 020-039 90 90 90 159 159 159 F F F 2 040-059 88 88 88 157 157 157 F F F 3 060-079 88 88 88 157 157 157 F F F 4 080-099 90 90 90 159 159 159 F F F 5 100-119 88 88 88 157 157 157 F F F 6 120-139 91 91 91 160 160 160 F F F 7 140-159 88 88 88 157 157 157 F F F 8 160-179 88 88 88 157 157 157 F F F 9 180-199 90 90 90 158 158 158 F F F 10 200-219 90 90 90 158 158 158 F F F 11 220-239 90 90 90 159 159 159 F F F 12 240-259 90 90 90 158 158 158 F F F 13 260-279 90 90 90 159 159 159 F F F 14 280-299 88 88 88 157 157 157 F F F 15 300-319 86 86 86 155 155 155 F F F 16 320-339 76 76 76 142 142 142 F F F 17 340-359 88 88 88 157 157 157 F F F 18 360-379 90 90 90 158 158 158 F F F 19 380-399 90 90 90 158 158 158 F F F 20 400-419 91 91 91 160 160 160 F F F 21 420-439 91 91 91 160 160 160 F F F 22 440-459 91 91 91 160 160 160 F F F 23 460-479 85 85 85 154 154 154 F F F 24 480-499 82 82 82 150 150 150 F F F 25 500-519 88 88 88 157 157 157 F F F 26 520-539 88 88 88 157 157 157 F F F 27 540-559 88 88 88 157 157 157 F F F 28 560-579 88 88 88 157 157 157 F F F 29 580-599 88 88 88 157 157 157 F F F Max gen: 91, Max flush: 160 Begin loop 46 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.276485557Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.284624693Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.284923407Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.284953891Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.28496266Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.284974298Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501} {"msg":"Crucible 928c86b3-a11c-4334-8e21-06514a9fe80f has session id: 0ba5d25c-167e-4046-a247-4e6f8cb91e35","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285303706Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"Upstairs opts: Upstairs UUID: 928c86b3-a11c-4334-8e21-06514a9fe80f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285328271Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"Crucible stats registered with UUID: 928c86b3-a11c-4334-8e21-06514a9fe80f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285338149Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:34.285347028Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285487951Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"io task","client":"1","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285504688Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"io task","client":"0","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285514646Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"io task","client":"2","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285670087Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"task":"crutest"} {"msg":"The guest has requested activation with gen:92","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285694113Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501} {"msg":"928c86b3-a11c-4334-8e21-06514a9fe80f active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285719528Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285728836Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"client":"0","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285740094Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"client":"1","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285749742Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"client":"2","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285759301Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"io task","client":"2","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285769029Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"io task","client":"0","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285778897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"io task","client":"1","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"ds_connection connected from Ok(127.0.0.1:36883)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.28581447Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"io task","client":"0","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"ds_connection connected from Ok(127.0.0.1:36253)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285843495Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"io task","client":"2","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"ds_connection connected from Ok(127.0.0.1:34841)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.285943196Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"io task","client":"1","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.286484495Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"client":"2","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.286503791Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"client":"1","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.286514479Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"client":"0","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.286794287Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.28682887Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"[0]R flush_numbers[0..12]: [159, 159, 157, 157, 159, 157, 160, 157, 157, 158, 158, 159]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.286914954Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"downstairs","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"[0]R generation[0..12]: [90, 90, 88, 88, 90, 88, 91, 88, 88, 90, 90, 90]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.286926082Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"downstairs","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.28693914Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"downstairs","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"[1]R flush_numbers[0..12]: [159, 159, 157, 157, 159, 157, 160, 157, 157, 158, 158, 159]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.286949658Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"downstairs","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"[1]R generation[0..12]: [90, 90, 88, 88, 90, 88, 91, 88, 88, 90, 90, 90]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.286959866Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"downstairs","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.286970624Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"downstairs","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"[2]R flush_numbers[0..12]: [159, 159, 157, 157, 159, 157, 160, 157, 157, 158, 158, 159]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.286981122Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"downstairs","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"[2]R generation[0..12]: [90, 90, 88, 88, 90, 88, 91, 88, 88, 90, 90, 90]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.28699134Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"downstairs","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287002008Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"downstairs","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"Max found gen is 92","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287011436Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"downstairs","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"Generation requested: 92 >= found:92","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287020984Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"downstairs","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"Next flush: 161","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287030313Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"downstairs","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287039571Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"":"downstairs","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287049169Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287057787Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"client":"0","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287067356Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"client":"1","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287078654Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"client":"2","":"downstairs","session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287088402Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.28709742Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"928c86b3-a11c-4334-8e21-06514a9fe80f is now active with session: 0ba5d25c-167e-4046-a247-4e6f8cb91e35","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287107078Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287115607Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"session_id":"0ba5d25c-167e-4046-a247-4e6f8cb91e35"} {"msg":"The guest has finished waiting for activation with:92","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287124525Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501} {"msg":"Activated sub_volume 928c86b3-a11c-4334-8e21-06514a9fe80f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287132393Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287140282Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.287482828Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"task":"crutest"} 01/30 Flush 02/30 Write block 128 len 32768 data: 12 13 16 14 14 15 14 12 03/30 Read block 164 len 8192 04/30 Read block 85 len 16384 05/30 Write block 391 len 16384 data: 9 10 11 10 06/30 Read block 420 len 4096 07/30 Read block 260 len 32768 08/30 Write block 245 len 32768 data: 7 7 8 8 9 9 10 11 09/30 Write block 476 len 12288 data: 5 5 5 10/30 Read block 36 len 8192 11/30 Write block 220 len 4096 data: 8 12/30 Read block 442 len 4096 13/30 Write block 409 len 40960 data: 3 4 4 4 3 6 6 6 4 4 14/30 Write block 243 len 12288 data: 7 6 8 15/30 Flush 16/30 Write block 3 len 20480 data: 3 3 3 6 7 17/30 Write block 543 len 8192 data: 9 9 18/30 Read block 540 len 28672 19/30 Read block 358 len 36864 20/30 Write block 214 len 32768 data: 14 14 13 11 11 8 9 7 21/30 Read block 304 len 8192 22/30 Write block 348 len 40960 data: 4 7 6 8 7 7 7 6 8 7 23/30 Read block 154 len 32768 24/30 Write block 545 len 28672 data: 9 8 7 7 7 7 9 25/30 Write block 335 len 40960 data: 9 9 7 8 7 6 6 8 8 8 26/30 Write block 246 len 16384 data: 8 9 9 10 27/30 Write block 61 len 32768 data: 11 12 8 7 7 8 8 7 28/30 Read block 386 len 28672 29/30 Write block 330 len 8192 data: 9 9 30/30 Write block 418 len 28672 data: 5 5 4 4 5 5 6 ---------------------------------------------------------------- Crucible gen:92 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 8 Done Done Done false 1002 Acked Read 2 Done Done Done false 1003 Acked Read 4 Done Done Done false 1004 Acked Write 4 Done Sent Done false 1005 Acked Read 1 Done Sent Done false 1006 Acked Read 8 Done Sent Done false 1007 Acked Write 8 Done Sent Done false 1008 Acked Write 3 Done Sent Done false 1009 Acked Read 2 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Read 1 Done Sent Done false 1012 Acked Write 10 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Read 9 Done Sent Done false 1019 Acked Write 8 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Write 10 Done Sent Done false 1022 Acked Read 8 Done Sent Done false 1023 Acked Write 7 Done Sent Done false 1024 Acked Write 10 Done Sent Done false 1025 Acked Write 4 Done Sent Done false 1026 Acked Write 8 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Write 2 Done Sent Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 26 1 28 Done 28 3 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1014)) Some(JobId(1000)) Some(JobId(1014)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:34.295716786Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:34.295915689Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1501,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 90 90 90 159 159 159 T F T <--- 1 020-039 90 90 90 159 159 159 F F F 2 040-059 88 88 88 157 157 157 F F F 3 060-079 88 88 88 157 157 157 T F T <--- 4 080-099 90 90 90 159 159 159 F F F 5 100-119 88 88 88 157 157 157 F F F 6 120-139 92 91 92 162 160 162 F T F <--- 7 140-159 88 88 88 157 157 157 F F F 8 160-179 88 88 88 157 157 157 F F F 9 180-199 90 90 90 158 158 158 F F F 10 200-219 90 90 90 158 158 158 T F T <--- 11 220-239 92 90 92 162 159 162 T F T <--- 12 240-259 92 90 92 162 158 162 T F T <--- 13 260-279 90 90 90 159 159 159 F F F 14 280-299 88 88 88 157 157 157 F F F 15 300-319 86 86 86 155 155 155 F F F 16 320-339 76 76 76 142 142 142 T F T <--- 17 340-359 88 88 88 157 157 157 T F T <--- 18 360-379 90 90 90 158 158 158 F F F 19 380-399 92 90 92 162 158 162 F F F <--- 20 400-419 92 91 92 162 160 162 T F T <--- 21 420-439 91 91 91 160 160 160 T F T <--- 22 440-459 91 91 91 160 160 160 F F F 23 460-479 92 85 92 162 154 162 F F F <--- 24 480-499 82 82 82 150 150 150 F F F 25 500-519 88 88 88 157 157 157 F F F 26 520-539 88 88 88 157 157 157 F F F 27 540-559 88 88 88 157 157 157 T F T <--- 28 560-579 88 88 88 157 157 157 F F F 29 580-599 88 88 88 157 157 157 F F F Max gen: 92, Max flush: 162 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 46 Verifying data now No change after verify Loop: 46 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 93 93 93 163 163 163 F F F 1 020-039 90 90 90 159 159 159 F F F 2 040-059 88 88 88 157 157 157 F F F 3 060-079 93 93 93 163 163 163 F F F 4 080-099 90 90 90 159 159 159 F F F 5 100-119 88 88 88 157 157 157 F F F 6 120-139 92 92 92 162 162 162 F F F 7 140-159 88 88 88 157 157 157 F F F 8 160-179 88 88 88 157 157 157 F F F 9 180-199 90 90 90 158 158 158 F F F 10 200-219 93 93 93 163 163 163 F F F 11 220-239 93 93 93 163 163 163 F F F 12 240-259 93 93 93 163 163 163 F F F 13 260-279 90 90 90 159 159 159 F F F 14 280-299 88 88 88 157 157 157 F F F 15 300-319 86 86 86 155 155 155 F F F 16 320-339 93 93 93 163 163 163 F F F 17 340-359 93 93 93 163 163 163 F F F 18 360-379 90 90 90 158 158 158 F F F 19 380-399 92 92 92 162 162 162 F F F 20 400-419 93 93 93 163 163 163 F F F 21 420-439 93 93 93 163 163 163 F F F 22 440-459 91 91 91 160 160 160 F F F 23 460-479 92 92 92 162 162 162 F F F 24 480-499 82 82 82 150 150 150 F F F 25 500-519 88 88 88 157 157 157 F F F 26 520-539 88 88 88 157 157 157 F F F 27 540-559 93 93 93 163 163 163 F F F 28 560-579 88 88 88 157 157 157 F F F 29 580-599 88 88 88 157 157 157 F F F Max gen: 93, Max flush: 163 Begin loop 47 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.549201644Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:38.556639321Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.556702139Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.563803639Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564102943Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564131818Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564142606Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564154033Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509} {"msg":"Crucible a0fa98ba-adc1-4322-bf71-acf8710ce9a8 has session id: e34e34d0-4919-47be-a4fd-d567449bfb5d","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.56449325Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"Upstairs opts: Upstairs UUID: a0fa98ba-adc1-4322-bf71-acf8710ce9a8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.56454369Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"Crucible stats registered with UUID: a0fa98ba-adc1-4322-bf71-acf8710ce9a8","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564554069Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:38.564563017Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564710289Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"io task","client":"1","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564726796Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"io task","client":"2","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564736754Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"io task","client":"0","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564885326Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"task":"crutest"} {"msg":"The guest has requested activation with gen:94","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564897874Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509} {"msg":"a0fa98ba-adc1-4322-bf71-acf8710ce9a8 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564915721Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564924449Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"client":"0","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564933977Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"client":"1","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564943376Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"client":"2","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564952764Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"io task","client":"2","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564964112Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"io task","client":"0","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.564974Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"io task","client":"1","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"ds_connection connected from Ok(127.0.0.1:43102)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.565021781Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"io task","client":"2","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"ds_connection connected from Ok(127.0.0.1:41638)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.565032599Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"io task","client":"1","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"ds_connection connected from Ok(127.0.0.1:45411)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.565042867Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"io task","client":"0","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.565729678Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"client":"2","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.565751904Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"client":"0","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.565769151Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"client":"1","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566045179Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566054697Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"[0]R flush_numbers[0..12]: [163, 159, 157, 163, 159, 157, 162, 157, 157, 158, 163, 163]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.56609586Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"downstairs","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"[0]R generation[0..12]: [93, 90, 88, 93, 90, 88, 92, 88, 88, 90, 93, 93]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566106878Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"downstairs","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566117716Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"downstairs","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"[1]R flush_numbers[0..12]: [163, 159, 157, 163, 159, 157, 162, 157, 157, 158, 163, 163]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566128514Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"downstairs","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"[1]R generation[0..12]: [93, 90, 88, 93, 90, 88, 92, 88, 88, 90, 93, 93]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566140761Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"downstairs","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566151649Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"downstairs","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"[2]R flush_numbers[0..12]: [163, 159, 157, 163, 159, 157, 162, 157, 157, 158, 163, 163]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566162007Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"downstairs","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"[2]R generation[0..12]: [93, 90, 88, 93, 90, 88, 92, 88, 88, 90, 93, 93]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566172285Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"downstairs","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566183103Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"downstairs","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"Max found gen is 94","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566192472Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"downstairs","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"Generation requested: 94 >= found:94","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.56620186Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"downstairs","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"Next flush: 164","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566210968Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"downstairs","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566220246Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"":"downstairs","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566230165Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566238953Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"client":"0","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566248731Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"client":"1","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566258559Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"client":"2","":"downstairs","session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566268507Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566277486Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"a0fa98ba-adc1-4322-bf71-acf8710ce9a8 is now active with session: e34e34d0-4919-47be-a4fd-d567449bfb5d","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566288324Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566296912Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"session_id":"e34e34d0-4919-47be-a4fd-d567449bfb5d"} {"msg":"The guest has finished waiting for activation with:94","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.56630581Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509} {"msg":"Activated sub_volume a0fa98ba-adc1-4322-bf71-acf8710ce9a8","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566313709Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566321847Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.566642457Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"task":"crutest"} 01/30 Write block 14 len 20480 data: 8 9 10 8 10 02/30 Write block 64 len 20480 data: 8 8 9 9 8 03/30 Read block 304 len 4096 04/30 Write block 561 len 28672 data: 9 10 10 8 8 7 6 05/30 Flush 06/30 Write block 33 len 12288 data: 11 9 8 07/30 Read block 370 len 16384 08/30 Read block 288 len 40960 09/30 Read block 219 len 36864 10/30 Write block 494 len 8192 data: 7 7 11/30 Write block 30 len 12288 data: 12 12 13 12/30 Write block 43 len 12288 data: 8 7 8 13/30 Flush 14/30 Write block 416 len 4096 data: 7 15/30 Read block 83 len 36864 16/30 Read block 60 len 36864 17/30 Write block 147 len 28672 data: 8 5 4 3 4 5 5 18/30 Read block 102 len 40960 19/30 Write block 6 len 32768 data: 7 8 7 6 9 11 11 9 20/30 Write block 233 len 40960 data: 7 8 7 8 8 8 4 5 7 7 21/30 Write block 406 len 36864 data: 8 6 5 4 5 5 5 4 7 22/30 Read block 39 len 40960 23/30 Flush 24/30 Write block 404 len 12288 data: 10 7 9 25/30 Write block 312 len 24576 data: 8 8 9 9 10 11 26/30 Write block 540 len 8192 data: 5 6 27/30 Read block 21 len 4096 28/30 Read block 583 len 20480 29/30 Read block 181 len 32768 30/30 Write block 261 len 40960 data: 9 9 8 6 5 6 9 7 8 7 ---------------------------------------------------------------- Crucible gen:94 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 5 Sent Done Done false 1002 Acked Read 1 Sent Done Done false 1003 Acked Write 7 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Write 3 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Read 10 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 1 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Read 9 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Read 10 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Write 10 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Read 10 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Read 5 Sent Done Done false 1028 Acked Read 8 Sent Done Done false 1029 Acked Write 10 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 1 31 Done 1 29 29 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:38.5752255Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:38.575399767Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1509,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 93 94 94 163 166 166 T F F <--- 1 020-039 90 94 94 159 165 165 F F F <--- 2 040-059 88 94 94 157 165 165 F F F <--- 3 060-079 93 94 94 163 164 164 F F F <--- 4 080-099 90 90 90 159 159 159 F F F 5 100-119 88 88 88 157 157 157 F F F 6 120-139 92 92 92 162 162 162 F F F 7 140-159 88 94 94 157 166 166 F F F <--- 8 160-179 88 88 88 157 157 157 F F F 9 180-199 90 90 90 158 158 158 F F F 10 200-219 93 93 93 163 163 163 F F F 11 220-239 93 94 94 163 166 166 F F F <--- 12 240-259 93 94 94 163 166 166 F F F <--- 13 260-279 90 90 90 159 159 159 F T T <--- 14 280-299 88 88 88 157 157 157 F F F 15 300-319 86 86 86 155 155 155 F T T <--- 16 320-339 93 93 93 163 163 163 F F F 17 340-359 93 93 93 163 163 163 F F F 18 360-379 90 90 90 158 158 158 F F F 19 380-399 92 92 92 162 162 162 F F F 20 400-419 93 94 94 163 166 166 F T T <--- 21 420-439 93 93 93 163 163 163 F F F 22 440-459 91 91 91 160 160 160 F F F 23 460-479 92 92 92 162 162 162 F F F 24 480-499 82 94 94 150 165 165 F F F <--- 25 500-519 88 88 88 157 157 157 F F F 26 520-539 88 88 88 157 157 157 F F F 27 540-559 93 93 93 163 163 163 F T T <--- 28 560-579 88 94 94 157 164 164 F F F <--- 29 580-599 88 88 88 157 157 157 F F F Max gen: 94, Max flush: 166 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 47 Verifying data now No change after verify Loop: 47 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 94 94 94 166 166 166 F F F 1 020-039 94 94 94 165 165 165 F F F 2 040-059 94 94 94 165 165 165 F F F 3 060-079 94 94 94 164 164 164 F F F 4 080-099 90 90 90 159 159 159 F F F 5 100-119 88 88 88 157 157 157 F F F 6 120-139 92 92 92 162 162 162 F F F 7 140-159 94 94 94 166 166 166 F F F 8 160-179 88 88 88 157 157 157 F F F 9 180-199 90 90 90 158 158 158 F F F 10 200-219 93 93 93 163 163 163 F F F 11 220-239 94 94 94 166 166 166 F F F 12 240-259 94 94 94 166 166 166 F F F 13 260-279 95 95 95 167 167 167 F F F 14 280-299 88 88 88 157 157 157 F F F 15 300-319 95 95 95 167 167 167 F F F 16 320-339 93 93 93 163 163 163 F F F 17 340-359 93 93 93 163 163 163 F F F 18 360-379 90 90 90 158 158 158 F F F 19 380-399 92 92 92 162 162 162 F F F 20 400-419 95 95 95 167 167 167 F F F 21 420-439 93 93 93 163 163 163 F F F 22 440-459 91 91 91 160 160 160 F F F 23 460-479 92 92 92 162 162 162 F F F 24 480-499 94 94 94 165 165 165 F F F 25 500-519 88 88 88 157 157 157 F F F 26 520-539 88 88 88 157 157 157 F F F 27 540-559 95 95 95 167 167 167 F F F 28 560-579 94 94 94 164 164 164 F F F 29 580-599 88 88 88 157 157 157 F F F Max gen: 95, Max flush: 167 Begin loop 48 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.858066201Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.866369656Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.866692705Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.86672062Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.866729438Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.866741176Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517} {"msg":"Crucible 3dd0b1d0-4258-4b2a-96fb-6941eb65a895 has session id: 1db11180-d826-48e9-a551-d1946e695955","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867114826Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"Upstairs opts: Upstairs UUID: 3dd0b1d0-4258-4b2a-96fb-6941eb65a895, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867176824Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"Crucible stats registered with UUID: 3dd0b1d0-4258-4b2a-96fb-6941eb65a895","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867189242Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:42.86719816Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867336785Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"io task","client":"0","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867353361Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"io task","client":"1","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.86736331Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"io task","client":"2","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867525249Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"task":"crutest"} {"msg":"The guest has requested activation with gen:96","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867540846Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517} {"msg":"3dd0b1d0-4258-4b2a-96fb-6941eb65a895 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.86757453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867583858Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"client":"0","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867593486Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"client":"1","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867603035Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"client":"2","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867612413Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"io task","client":"2","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867622221Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"io task","client":"0","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867632079Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"io task","client":"1","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"ds_connection connected from Ok(127.0.0.1:34938)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867677321Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"io task","client":"2","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"ds_connection connected from Ok(127.0.0.1:47094)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867688299Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"io task","client":"1","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"ds_connection connected from Ok(127.0.0.1:49993)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.867723992Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"io task","client":"0","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868366692Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"client":"1","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868395816Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"client":"0","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868557246Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"client":"2","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868680963Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868719486Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"[0]R flush_numbers[0..12]: [166, 165, 165, 164, 159, 157, 162, 166, 157, 158, 163, 166]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868881495Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"downstairs","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"[0]R generation[0..12]: [94, 94, 94, 94, 90, 88, 92, 94, 88, 90, 93, 94]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868892563Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"downstairs","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868903501Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"downstairs","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"[1]R flush_numbers[0..12]: [166, 165, 165, 164, 159, 157, 162, 166, 157, 158, 163, 166]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868913929Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"downstairs","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"[1]R generation[0..12]: [94, 94, 94, 94, 90, 88, 92, 94, 88, 90, 93, 94]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868924297Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"downstairs","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868935115Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"downstairs","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"[2]R flush_numbers[0..12]: [166, 165, 165, 164, 159, 157, 162, 166, 157, 158, 163, 166]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868945643Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"downstairs","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"[2]R generation[0..12]: [94, 94, 94, 94, 90, 88, 92, 94, 88, 90, 93, 94]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868955811Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"downstairs","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868968359Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"downstairs","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"Max found gen is 96","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868977787Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"downstairs","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"Generation requested: 96 >= found:96","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868987405Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"downstairs","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"Next flush: 168","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.868996564Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"downstairs","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.869006202Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"":"downstairs","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.86901598Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.869024868Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"client":"0","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.869034387Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"client":"1","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.869043855Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"client":"2","":"downstairs","session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.869053563Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.869062591Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"3dd0b1d0-4258-4b2a-96fb-6941eb65a895 is now active with session: 1db11180-d826-48e9-a551-d1946e695955","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.869072459Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-07-20T22:59:42.869081098Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"session_id":"1db11180-d826-48e9-a551-d1946e695955"} {"msg":"The guest has finished waiting for activation with:96","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.869095595Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517} {"msg":"Activated sub_volume 3dd0b1d0-4258-4b2a-96fb-6941eb65a895","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.869103494Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.869111472Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"task":"crutest"} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.869353547Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"task":"crutest"} 01/30 Write block 109 len 28672 data: 12 14 12 12 11 10 9 02/30 Read block 130 len 16384 03/30 Write block 89 len 4096 data: 6 04/30 Read block 412 len 12288 05/30 Read block 12 len 16384 06/30 Read block 379 len 28672 07/30 Write block 58 len 20480 data: 10 12 12 12 13 08/30 Write block 260 len 20480 data: 9 10 10 9 7 09/30 Write block 421 len 16384 data: 5 6 6 7 10/30 Read block 301 len 24576 11/30 Read block 488 len 16384 12/30 Read block 497 len 32768 13/30 Read block 403 len 20480 14/30 Write block 287 len 24576 data: 6 8 9 9 7 8 15/30 Write block 16 len 28672 data: 11 9 11 10 9 8 8 16/30 Write block 93 len 16384 data: 6 7 6 6 17/30 Write block 545 len 16384 data: 10 9 8 8 18/30 Read block 52 len 24576 19/30 Flush 20/30 Write block 22 len 40960 data: 9 9 8 8 8 6 8 10 13 13 21/30 Read block 51 len 36864 22/30 Read block 544 len 8192 23/30 Write block 577 len 28672 data: 8 10 9 8 9 9 7 24/30 Write block 151 len 8192 data: 5 6 25/30 Write block 419 len 16384 data: 6 5 6 7 26/30 Read block 97 len 12288 27/30 Write block 118 len 4096 data: 7 28/30 Read block 562 len 4096 29/30 Write block 89 len 4096 data: 7 30/30 Read block 372 len 36864 ---------------------------------------------------------------- Crucible gen:96 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 4 Done Done Done false 1002 Acked Write 1 Done Done Done false 1003 Acked Read 3 Done Done Sent false 1004 Acked Read 4 Done Done Sent false 1005 Acked Read 7 Done Done Sent false 1006 Acked Write 5 Done Done Sent false 1007 Acked Write 5 Done Done Sent false 1008 Acked Write 4 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Read 8 Done Done Sent false 1012 Acked Read 5 Done Done Sent false 1013 Acked Write 6 Done Done Sent false 1014 Acked Write 7 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Write 4 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 10 Done Done Sent false 1020 Acked Read 9 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Write 7 Done Done Sent false 1023 Acked Write 2 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Write 1 Done Done Sent false 1027 Acked Read 1 Done Done Sent false 1028 Acked Write 1 Done Done Sent false 1029 Acked Read 9 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 27 27 Done 30 30 3 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1018)) Some(JobId(1018)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:42.877868562Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:42.878070014Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1517,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 96 96 94 168 168 166 F F F <--- 1 020-039 96 96 94 168 168 165 T T F <--- 2 040-059 96 96 94 168 168 165 F F F <--- 3 060-079 96 96 94 168 168 164 F F F <--- 4 080-099 96 96 90 168 168 159 T T T <--- 5 100-119 96 96 88 168 168 157 T T T <--- 6 120-139 92 92 92 162 162 162 F F F 7 140-159 94 94 94 166 166 166 T T F <--- 8 160-179 88 88 88 157 157 157 F F F 9 180-199 90 90 90 158 158 158 F F F 10 200-219 93 93 93 163 163 163 F F F 11 220-239 94 94 94 166 166 166 F F F 12 240-259 94 94 94 166 166 166 F F F 13 260-279 96 96 95 168 168 167 F F F <--- 14 280-299 96 96 88 168 168 157 F F F <--- 15 300-319 95 95 95 167 167 167 F F F 16 320-339 93 93 93 163 163 163 F F F 17 340-359 93 93 93 163 163 163 F F F 18 360-379 90 90 90 158 158 158 F F F 19 380-399 92 92 92 162 162 162 F F F 20 400-419 95 95 95 167 167 167 T T F <--- 21 420-439 96 96 93 168 168 163 T T F <--- 22 440-459 91 91 91 160 160 160 F F F 23 460-479 92 92 92 162 162 162 F F F 24 480-499 94 94 94 165 165 165 F F F 25 500-519 88 88 88 157 157 157 F F F 26 520-539 88 88 88 157 157 157 F F F 27 540-559 96 96 95 168 168 167 F F F <--- 28 560-579 94 94 94 164 164 164 T T F <--- 29 580-599 88 88 88 157 157 157 T T F <--- Max gen: 96, Max flush: 168 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 48 Verifying data now No change after verify Loop: 48 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 96 96 96 168 168 168 F F F 1 020-039 97 97 97 169 169 169 F F F 2 040-059 96 96 96 168 168 168 F F F 3 060-079 96 96 96 168 168 168 F F F 4 080-099 97 97 97 169 169 169 F F F 5 100-119 97 97 97 169 169 169 F F F 6 120-139 92 92 92 162 162 162 F F F 7 140-159 97 97 97 169 169 169 F F F 8 160-179 88 88 88 157 157 157 F F F 9 180-199 90 90 90 158 158 158 F F F 10 200-219 93 93 93 163 163 163 F F F 11 220-239 94 94 94 166 166 166 F F F 12 240-259 94 94 94 166 166 166 F F F 13 260-279 96 96 96 168 168 168 F F F 14 280-299 96 96 96 168 168 168 F F F 15 300-319 95 95 95 167 167 167 F F F 16 320-339 93 93 93 163 163 163 F F F 17 340-359 93 93 93 163 163 163 F F F 18 360-379 90 90 90 158 158 158 F F F 19 380-399 92 92 92 162 162 162 F F F 20 400-419 97 97 97 169 169 169 F F F 21 420-439 97 97 97 169 169 169 F F F 22 440-459 91 91 91 160 160 160 F F F 23 460-479 92 92 92 162 162 162 F F F 24 480-499 94 94 94 165 165 165 F F F 25 500-519 88 88 88 157 157 157 F F F 26 520-539 88 88 88 157 157 157 F F F 27 540-559 96 96 96 168 168 168 F F F 28 560-579 97 97 97 169 169 169 F F F 29 580-599 97 97 97 169 169 169 F F F Max gen: 97, Max flush: 169 Begin loop 49 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.133350577Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.141532305Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.141842667Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.14187924Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.141888708Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.141900406Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525} {"msg":"Crucible fc06cce1-5f12-4fb3-9888-9ef5dcaaa3df has session id: 968563fd-6ab7-4fe0-a229-cd104b08bf04","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142215847Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"Upstairs opts: Upstairs UUID: fc06cce1-5f12-4fb3-9888-9ef5dcaaa3df, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142265067Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"Crucible stats registered with UUID: fc06cce1-5f12-4fb3-9888-9ef5dcaaa3df","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142276645Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:47.142285954Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142433096Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"io task","client":"1","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.14246494Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"io task","client":"0","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142476178Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"io task","client":"2","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142606084Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"task":"crutest"} {"msg":"The guest has requested activation with gen:98","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142617991Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525} {"msg":"fc06cce1-5f12-4fb3-9888-9ef5dcaaa3df active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142631309Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142640017Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"client":"0","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142649585Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"client":"1","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142659124Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"client":"2","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142668542Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"io task","client":"2","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.14267838Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"io task","client":"0","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142688288Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"io task","client":"1","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"ds_connection connected from Ok(127.0.0.1:57556)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.14278616Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"io task","client":"1","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"ds_connection connected from Ok(127.0.0.1:60619)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142798348Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"io task","client":"0","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"ds_connection connected from Ok(127.0.0.1:35497)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.142808596Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"io task","client":"2","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143546987Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"client":"2","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143564014Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"client":"0","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143575762Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"client":"1","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143861918Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143871487Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"[0]R flush_numbers[0..12]: [168, 169, 168, 168, 169, 169, 162, 169, 157, 158, 163, 166]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143911529Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"downstairs","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"[0]R generation[0..12]: [96, 97, 96, 96, 97, 97, 92, 97, 88, 90, 93, 94]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143922557Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"downstairs","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143933405Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"downstairs","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"[1]R flush_numbers[0..12]: [168, 169, 168, 168, 169, 169, 162, 169, 157, 158, 163, 166]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143944083Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"downstairs","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"[1]R generation[0..12]: [96, 97, 96, 96, 97, 97, 92, 97, 88, 90, 93, 94]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143954221Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"downstairs","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143964949Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"downstairs","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"[2]R flush_numbers[0..12]: [168, 169, 168, 168, 169, 169, 162, 169, 157, 158, 163, 166]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143975457Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"downstairs","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"[2]R generation[0..12]: [96, 97, 96, 96, 97, 97, 92, 97, 88, 90, 93, 94]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143985775Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"downstairs","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.143996633Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"downstairs","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"Max found gen is 98","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144006101Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"downstairs","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"Generation requested: 98 >= found:98","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.14401583Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"downstairs","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"Next flush: 170","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144025178Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"downstairs","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144034476Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"":"downstairs","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144045344Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144054052Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"client":"0","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144063551Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"client":"1","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144073149Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"client":"2","":"downstairs","session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144082807Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144091795Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"fc06cce1-5f12-4fb3-9888-9ef5dcaaa3df is now active with session: 968563fd-6ab7-4fe0-a229-cd104b08bf04","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144101873Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144110582Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"session_id":"968563fd-6ab7-4fe0-a229-cd104b08bf04"} {"msg":"The guest has finished waiting for activation with:98","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.14411964Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525} {"msg":"Activated sub_volume fc06cce1-5f12-4fb3-9888-9ef5dcaaa3df","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144127449Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144135507Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.144436991Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"task":"crutest"} 01/30 Write block 587 len 36864 data: 9 10 11 12 11 12 14 13 11 02/30 Write block 338 len 36864 data: 9 8 7 7 9 9 9 8 7 03/30 Read block 551 len 12288 04/30 Write block 357 len 16384 data: 8 4 4 4 05/30 Flush 06/30 Flush 07/30 Write block 85 len 40960 data: 9 8 6 6 8 6 7 5 7 8 08/30 Write block 332 len 36864 data: 11 9 9 10 10 8 10 9 8 09/30 Flush 10/30 Write block 5 len 12288 data: 4 8 9 11/30 Write block 11 len 20480 data: 12 12 10 9 10 12/30 Write block 141 len 4096 data: 14 13/30 Flush 14/30 Write block 260 len 8192 data: 10 11 15/30 Read block 216 len 4096 16/30 Write block 229 len 40960 data: 6 7 6 8 8 9 8 9 9 9 17/30 Read block 198 len 8192 18/30 Read block 493 len 12288 19/30 Write block 565 len 28672 data: 9 8 7 6 6 5 5 20/30 Write block 522 len 20480 data: 9 9 11 11 9 21/30 Read block 169 len 32768 22/30 Read block 8 len 36864 23/30 Read block 158 len 12288 24/30 Read block 154 len 24576 25/30 Read block 4 len 24576 26/30 Write block 68 len 24576 data: 9 6 5 5 6 5 27/30 Write block 121 len 16384 data: 9 8 10 12 28/30 Read block 82 len 20480 29/30 Write block 213 len 20480 data: 12 15 15 14 12 30/30 Read block 62 len 28672 ---------------------------------------------------------------- Crucible gen:98 GIO:true work queues: Upstairs:0 downstairs:21 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1009 Acked Write 3 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 2 Done Sent Done false 1014 Acked Read 1 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Read 3 Done Sent Done false 1018 Acked Write 7 Done Sent Done false 1019 Acked Write 5 Done Sent Done false 1020 Acked Read 8 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Read 3 Done Sent Done false 1023 Acked Read 6 Done Sent Done false 1024 Acked Read 6 Done Sent Done false 1025 Acked Write 6 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Write 5 Done Sent Done false 1029 Acked Read 7 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 21 1 22 Done 21 0 20 41 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) Some(JobId(1008)) Some(JobId(1012)) Downstairs last five completed: 1008 1007 1006 1005 1004 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:47.152964404Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:47.15314439Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1525,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 98 96 98 173 168 173 F F F <--- 1 020-039 97 97 97 169 169 169 F F F 2 040-059 96 96 96 168 168 168 F F F 3 060-079 96 96 96 168 168 168 T F T <--- 4 080-099 98 98 98 172 172 172 F F F 5 100-119 97 97 97 169 169 169 F F F 6 120-139 92 92 92 162 162 162 T F T <--- 7 140-159 98 97 98 173 169 173 F F F <--- 8 160-179 88 88 88 157 157 157 F F F 9 180-199 90 90 90 158 158 158 F F F 10 200-219 93 93 93 163 163 163 T F T <--- 11 220-239 94 94 94 166 166 166 T F T <--- 12 240-259 94 94 94 166 166 166 F F F 13 260-279 96 96 96 168 168 168 T F T <--- 14 280-299 96 96 96 168 168 168 F F F 15 300-319 95 95 95 167 167 167 F F F 16 320-339 98 98 98 172 172 172 F F F 17 340-359 98 98 98 172 172 172 F F F 18 360-379 98 98 98 170 170 170 F F F 19 380-399 92 92 92 162 162 162 F F F 20 400-419 97 97 97 169 169 169 F F F 21 420-439 97 97 97 169 169 169 F F F 22 440-459 91 91 91 160 160 160 F F F 23 460-479 92 92 92 162 162 162 F F F 24 480-499 94 94 94 165 165 165 F F F 25 500-519 88 88 88 157 157 157 F F F 26 520-539 88 88 88 157 157 157 T F T <--- 27 540-559 96 96 96 168 168 168 F F F 28 560-579 97 97 97 169 169 169 T F T <--- 29 580-599 98 98 98 170 170 170 F F F Max gen: 98, Max flush: 173 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 49 Verifying data now No change after verify Loop: 49 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 98 98 98 173 173 173 F F F 1 020-039 97 97 97 169 169 169 F F F 2 040-059 96 96 96 168 168 168 F F F 3 060-079 99 99 99 174 174 174 F F F 4 080-099 98 98 98 172 172 172 F F F 5 100-119 97 97 97 169 169 169 F F F 6 120-139 99 99 99 174 174 174 F F F 7 140-159 98 98 98 173 173 173 F F F 8 160-179 88 88 88 157 157 157 F F F 9 180-199 90 90 90 158 158 158 F F F 10 200-219 99 99 99 174 174 174 F F F 11 220-239 99 99 99 174 174 174 F F F 12 240-259 94 94 94 166 166 166 F F F 13 260-279 99 99 99 174 174 174 F F F 14 280-299 96 96 96 168 168 168 F F F 15 300-319 95 95 95 167 167 167 F F F 16 320-339 98 98 98 172 172 172 F F F 17 340-359 98 98 98 172 172 172 F F F 18 360-379 98 98 98 170 170 170 F F F 19 380-399 92 92 92 162 162 162 F F F 20 400-419 97 97 97 169 169 169 F F F 21 420-439 97 97 97 169 169 169 F F F 22 440-459 91 91 91 160 160 160 F F F 23 460-479 92 92 92 162 162 162 F F F 24 480-499 94 94 94 165 165 165 F F F 25 500-519 88 88 88 157 157 157 F F F 26 520-539 99 99 99 174 174 174 F F F 27 540-559 96 96 96 168 168 168 F F F 28 560-579 99 99 99 174 174 174 F F F 29 580-599 98 98 98 170 170 170 F F F Max gen: 99, Max flush: 174 Begin loop 50 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.399920905Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.408113151Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.40843939Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.408462316Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.408471214Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.408484761Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533} {"msg":"Crucible 4823b589-6e8d-4309-b514-023885b66776 has session id: 5436da4d-3bf2-42f5-857b-5f01b1247cde","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.408783665Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"Upstairs opts: Upstairs UUID: 4823b589-6e8d-4309-b514-023885b66776, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.40881199Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"Crucible stats registered with UUID: 4823b589-6e8d-4309-b514-023885b66776","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.408823328Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:51.408832536Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.408968841Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"io task","client":"1","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.409009663Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"io task","client":"0","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.409020601Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"io task","client":"2","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.409146767Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"task":"crutest"} {"msg":"The guest has requested activation with gen:100","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.409161375Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533} {"msg":"4823b589-6e8d-4309-b514-023885b66776 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.40918492Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.409194319Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"client":"0","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.409203887Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"client":"1","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.409213425Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"client":"2","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.409223083Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"io task","client":"2","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.409234661Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"io task","client":"0","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.409268405Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"io task","client":"1","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"ds_connection connected from Ok(127.0.0.1:42538)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.409298969Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"io task","client":"0","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"ds_connection connected from Ok(127.0.0.1:41322)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.409313586Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"io task","client":"2","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"ds_connection connected from Ok(127.0.0.1:65152)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.409333912Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"io task","client":"1","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410011435Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"client":"0","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410037431Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"client":"1","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410151049Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"client":"2","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410321827Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410331416Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"[0]R flush_numbers[0..12]: [173, 169, 168, 174, 172, 169, 174, 173, 157, 158, 174, 174]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410477698Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"downstairs","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"[0]R generation[0..12]: [98, 97, 96, 99, 98, 97, 99, 98, 88, 90, 99, 99]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410500604Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"downstairs","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410512232Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"downstairs","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"[1]R flush_numbers[0..12]: [173, 169, 168, 174, 172, 169, 174, 173, 157, 158, 174, 174]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.4105231Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"downstairs","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"[1]R generation[0..12]: [98, 97, 96, 99, 98, 97, 99, 98, 88, 90, 99, 99]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410535717Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"downstairs","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410546625Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"downstairs","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"[2]R flush_numbers[0..12]: [173, 169, 168, 174, 172, 169, 174, 173, 157, 158, 174, 174]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410557213Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"downstairs","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"[2]R generation[0..12]: [98, 97, 96, 99, 98, 97, 99, 98, 88, 90, 99, 99]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410567331Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"downstairs","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410578189Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"downstairs","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"Max found gen is 100","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410587558Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"downstairs","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"Generation requested: 100 >= found:100","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410597236Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"downstairs","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"Next flush: 175","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410606494Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"downstairs","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410615812Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"":"downstairs","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.41062568Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410634659Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"client":"0","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410644187Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"client":"1","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410653715Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"client":"2","":"downstairs","session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410663483Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410672442Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"4823b589-6e8d-4309-b514-023885b66776 is now active with session: 5436da4d-3bf2-42f5-857b-5f01b1247cde","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.41068246Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410692698Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"session_id":"5436da4d-3bf2-42f5-857b-5f01b1247cde"} {"msg":"The guest has finished waiting for activation with:100","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410701696Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533} {"msg":"Activated sub_volume 4823b589-6e8d-4309-b514-023885b66776","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410709715Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.410717733Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.411038923Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"task":"crutest"} 01/30 Write block 534 len 32768 data: 6 6 5 5 5 5 6 7 02/30 Write block 15 len 8192 data: 11 12 03/30 Write block 579 len 16384 data: 10 9 10 10 04/30 Read block 268 len 16384 05/30 Write block 592 len 20480 data: 13 15 14 12 9 06/30 Write block 279 len 28672 data: 7 6 5 4 4 4 5 07/30 Read block 539 len 36864 08/30 Read block 422 len 36864 09/30 Write block 337 len 8192 data: 9 11 10/30 Write block 517 len 40960 data: 8 7 7 7 9 10 10 12 12 10 11/30 Read block 74 len 4096 12/30 Read block 588 len 28672 13/30 Read block 398 len 28672 14/30 Write block 415 len 32768 data: 7 8 5 6 7 6 7 8 15/30 Write block 43 len 40960 data: 9 8 9 7 5 6 6 5 5 4 16/30 Write block 452 len 36864 data: 7 7 8 7 10 8 5 6 5 17/30 Write block 302 len 28672 data: 11 13 14 11 11 12 11 18/30 Read block 94 len 8192 19/30 Write block 433 len 16384 data: 3 3 5 4 20/30 Write block 434 len 8192 data: 4 6 21/30 Read block 493 len 36864 22/30 Read block 61 len 16384 23/30 Write block 458 len 12288 data: 6 7 6 24/30 Read block 490 len 24576 25/30 Flush 26/30 Write block 7 len 40960 data: 10 8 7 10 13 13 11 10 12 13 27/30 Read block 229 len 36864 28/30 Read block 459 len 8192 29/30 Read block 62 len 4096 30/30 Read block 303 len 40960 ---------------------------------------------------------------- Crucible gen:100 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Write 4 Done Done Done false 1003 Acked Read 4 Done Done Done false 1004 Acked Write 5 Done Done Done false 1005 Acked Write 7 Done Done Done false 1006 Acked Read 9 Done Done Done false 1007 Acked Read 9 Done Done Done false 1008 Acked Write 2 Done Done Done false 1009 Acked Write 10 Done Done Done false 1010 Acked Read 1 Done Done Sent false 1011 Acked Read 7 Done Done Sent false 1012 Acked Read 7 Done Done Sent false 1013 Acked Write 8 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Write 9 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Write 4 Done Done Sent false 1019 Acked Write 2 Done Done Sent false 1020 Acked Read 9 Done Done Sent false 1021 Acked Read 4 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Read 6 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 10 Done Done Sent false 1026 Acked Read 9 Done Done Sent false 1027 Acked Read 2 Done Done Sent false 1028 Acked Read 1 Done Done Sent false 1029 Acked Read 10 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 20 21 Done 29 30 10 69 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:51.419751701Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:51.419937686Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1533,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 100 100 98 175 175 173 T T T <--- 1 020-039 97 97 97 169 169 169 F F F 2 040-059 100 100 96 175 175 168 F F F <--- 3 060-079 99 99 99 174 174 174 F F F 4 080-099 98 98 98 172 172 172 F F F 5 100-119 97 97 97 169 169 169 F F F 6 120-139 99 99 99 174 174 174 F F F 7 140-159 98 98 98 173 173 173 F F F 8 160-179 88 88 88 157 157 157 F F F 9 180-199 90 90 90 158 158 158 F F F 10 200-219 99 99 99 174 174 174 F F F 11 220-239 99 99 99 174 174 174 F F F 12 240-259 94 94 94 166 166 166 F F F 13 260-279 100 100 99 175 175 174 F F T <--- 14 280-299 100 100 96 175 175 168 F F T <--- 15 300-319 100 100 95 175 175 167 F F F <--- 16 320-339 100 100 98 175 175 172 F F T <--- 17 340-359 98 98 98 172 172 172 F F F 18 360-379 98 98 98 170 170 170 F F F 19 380-399 92 92 92 162 162 162 F F F 20 400-419 100 100 97 175 175 169 F F F <--- 21 420-439 100 100 97 175 175 169 F F F <--- 22 440-459 100 100 91 175 175 160 F F F <--- 23 460-479 100 100 92 175 175 162 F F F <--- 24 480-499 94 94 94 165 165 165 F F F 25 500-519 100 100 88 175 175 157 F F T <--- 26 520-539 100 100 99 175 175 174 F F T <--- 27 540-559 100 100 96 175 175 168 F F T <--- 28 560-579 100 100 99 175 175 174 F F T <--- 29 580-599 100 100 98 175 175 170 F F T <--- Max gen: 100, Max flush: 175 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 50 Verifying data now No change after verify Loop: 50 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 101 101 101 176 176 176 F F F 1 020-039 97 97 97 169 169 169 F F F 2 040-059 100 100 100 175 175 175 F F F 3 060-079 99 99 99 174 174 174 F F F 4 080-099 98 98 98 172 172 172 F F F 5 100-119 97 97 97 169 169 169 F F F 6 120-139 99 99 99 174 174 174 F F F 7 140-159 98 98 98 173 173 173 F F F 8 160-179 88 88 88 157 157 157 F F F 9 180-199 90 90 90 158 158 158 F F F 10 200-219 99 99 99 174 174 174 F F F 11 220-239 99 99 99 174 174 174 F F F 12 240-259 94 94 94 166 166 166 F F F 13 260-279 100 100 100 175 175 175 F F F 14 280-299 100 100 100 175 175 175 F F F 15 300-319 100 100 100 175 175 175 F F F 16 320-339 100 100 100 175 175 175 F F F 17 340-359 98 98 98 172 172 172 F F F 18 360-379 98 98 98 170 170 170 F F F 19 380-399 92 92 92 162 162 162 F F F 20 400-419 100 100 100 175 175 175 F F F 21 420-439 100 100 100 175 175 175 F F F 22 440-459 100 100 100 175 175 175 F F F 23 460-479 100 100 100 175 175 175 F F F 24 480-499 94 94 94 165 165 165 F F F 25 500-519 100 100 100 175 175 175 F F F 26 520-539 100 100 100 175 175 175 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 100 100 100 175 175 175 F F F 29 580-599 100 100 100 175 175 175 F F F Max gen: 101, Max flush: 176 Begin loop 51 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.670899908Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.679082665Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.679427571Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.679453196Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.679461714Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.679473132Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541} {"msg":"Crucible a75897f6-7df9-46dd-b2d6-e13f5279f262 has session id: 9493c512-32e1-4499-a818-9c3654c64f3f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.679866079Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"Upstairs opts: Upstairs UUID: a75897f6-7df9-46dd-b2d6-e13f5279f262, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.679890734Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"Crucible stats registered with UUID: a75897f6-7df9-46dd-b2d6-e13f5279f262","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.679902612Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:55.67991163Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.68007485Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"io task","client":"2","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680105164Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"io task","client":"0","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680116442Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"io task","client":"1","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680254146Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"task":"crutest"} {"msg":"The guest has requested activation with gen:102","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680271803Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541} {"msg":"a75897f6-7df9-46dd-b2d6-e13f5279f262 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680307036Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680316524Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"client":"0","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680326083Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"client":"1","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680335571Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"client":"2","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680344879Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"io task","client":"2","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680354897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"io task","client":"0","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680364825Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"io task","client":"1","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"ds_connection connected from Ok(127.0.0.1:41693)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680423734Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"io task","client":"1","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"ds_connection connected from Ok(127.0.0.1:48643)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680438422Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"io task","client":"2","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"ds_connection connected from Ok(127.0.0.1:50390)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.680450999Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"io task","client":"0","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681102297Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"client":"0","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.68119408Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"client":"2","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681205558Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"client":"1","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681387214Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.68151514Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"[0]R flush_numbers[0..12]: [176, 169, 175, 174, 172, 169, 174, 173, 157, 158, 174, 174]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681543065Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"downstairs","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"[0]R generation[0..12]: [101, 97, 100, 99, 98, 97, 99, 98, 88, 90, 99, 99]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681555102Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"downstairs","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.68156629Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"downstairs","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"[1]R flush_numbers[0..12]: [176, 169, 175, 174, 172, 169, 174, 173, 157, 158, 174, 174]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681576878Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"downstairs","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"[1]R generation[0..12]: [101, 97, 100, 99, 98, 97, 99, 98, 88, 90, 99, 99]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681587326Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"downstairs","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681598264Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"downstairs","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"[2]R flush_numbers[0..12]: [176, 169, 175, 174, 172, 169, 174, 173, 157, 158, 174, 174]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681609062Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"downstairs","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"[2]R generation[0..12]: [101, 97, 100, 99, 98, 97, 99, 98, 88, 90, 99, 99]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.68161945Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"downstairs","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681633008Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"downstairs","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"Max found gen is 102","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681642376Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"downstairs","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"Generation requested: 102 >= found:102","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681651984Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"downstairs","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"Next flush: 177","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681661552Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"downstairs","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681670861Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"":"downstairs","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681680549Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681689317Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"client":"0","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681698885Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"client":"1","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681708424Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"client":"2","":"downstairs","session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681718212Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.68172724Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"a75897f6-7df9-46dd-b2d6-e13f5279f262 is now active with session: 9493c512-32e1-4499-a818-9c3654c64f3f","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681737108Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681745697Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"session_id":"9493c512-32e1-4499-a818-9c3654c64f3f"} {"msg":"The guest has finished waiting for activation with:102","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681754705Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541} {"msg":"Activated sub_volume a75897f6-7df9-46dd-b2d6-e13f5279f262","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681762524Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.681770612Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.682114338Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"task":"crutest"} 01/30 Write block 511 len 24576 data: 6 6 6 7 8 8 02/30 Write block 260 len 16384 data: 11 12 11 10 03/30 Write block 11 len 40960 data: 14 14 12 11 13 14 10 12 11 10 04/30 Write block 575 len 24576 data: 7 7 9 11 11 10 05/30 Write block 205 len 4096 data: 5 06/30 Write block 397 len 28672 data: 11 10 8 5 6 8 9 07/30 Read block 101 len 36864 08/30 Read block 359 len 8192 09/30 Read block 573 len 36864 10/30 Flush 11/30 Write block 221 len 4096 data: 8 12/30 Write block 218 len 20480 data: 12 9 10 9 5 13/30 Write block 511 len 28672 data: 7 7 7 8 9 9 9 14/30 Read block 580 len 4096 15/30 Write block 170 len 24576 data: 4 4 6 6 8 6 16/30 Write block 227 len 16384 data: 5 6 7 8 17/30 Write block 437 len 16384 data: 5 5 6 7 18/30 Read block 361 len 4096 19/30 Read block 592 len 20480 20/30 Write block 416 len 28672 data: 9 6 7 8 7 8 9 21/30 Flush 22/30 Write block 519 len 24576 data: 8 8 10 11 11 13 23/30 Flush 24/30 Flush 25/30 Write block 164 len 8192 data: 6 7 26/30 Read block 399 len 4096 27/30 Write block 304 len 16384 data: 15 12 12 13 28/30 Write block 217 len 24576 data: 13 13 10 11 10 6 29/30 Read block 326 len 24576 30/30 Read block 386 len 32768 ---------------------------------------------------------------- Crucible gen:102 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Write 10 Done Done Done false 1003 Acked Write 6 Done Done Done false 1004 Acked Write 1 Done Done Done false 1005 Acked Write 7 Done Done Done false 1006 Acked Read 9 Done Sent Done false 1007 Acked Read 2 Done Sent Done false 1008 Acked Read 9 Done Sent Done false 1009 Acked Flush 0 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Write 5 Done Sent Done false 1012 Acked Write 7 Done Sent Done false 1013 Acked Read 1 Done Sent Done false 1014 Acked Write 6 Done Sent Done false 1015 Acked Write 4 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Read 5 Done Sent Done false 1019 Acked Write 7 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 2 Done Sent Done false 1025 Acked Read 1 Done Sent Done false 1026 Acked Write 4 Done Sent Done false 1027 Acked Write 6 Done Sent Done false 1028 Acked Read 6 Done Sent Done false 1029 Acked Read 8 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 24 0 25 Done 29 6 30 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:55.689915137Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:55.69011137Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1541,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 101 102 177 176 177 F T F <--- 1 020-039 102 97 102 177 169 177 F T F <--- 2 040-059 100 100 100 175 175 175 F F F 3 060-079 99 99 99 174 174 174 F F F 4 080-099 98 98 98 172 172 172 F F F 5 100-119 97 97 97 169 169 169 F F F 6 120-139 99 99 99 174 174 174 F F F 7 140-159 98 98 98 173 173 173 F F F 8 160-179 102 88 102 178 157 178 T F T <--- 9 180-199 90 90 90 158 158 158 F F F 10 200-219 102 99 102 178 174 178 T T T <--- 11 220-239 102 99 102 178 174 178 T F T <--- 12 240-259 94 94 94 166 166 166 F F F 13 260-279 102 100 102 177 175 177 F T F <--- 14 280-299 100 100 100 175 175 175 F F F 15 300-319 100 100 100 175 175 175 T F T <--- 16 320-339 100 100 100 175 175 175 F F F 17 340-359 98 98 98 172 172 172 F F F 18 360-379 98 98 98 170 170 170 F F F 19 380-399 102 92 102 177 162 177 F T F <--- 20 400-419 102 100 102 178 175 178 F T F <--- 21 420-439 102 100 102 178 175 178 F F F <--- 22 440-459 102 100 102 178 175 178 F F F <--- 23 460-479 100 100 100 175 175 175 F F F 24 480-499 94 94 94 165 165 165 F F F 25 500-519 102 100 102 179 175 179 F T F <--- 26 520-539 102 100 102 179 175 179 F F F <--- 27 540-559 100 100 100 175 175 175 F F F 28 560-579 102 100 102 177 175 177 F T F <--- 29 580-599 102 100 102 177 175 177 F T F <--- Max gen: 102, Max flush: 179 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 51 Verifying data now No change after verify Loop: 51 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 177 177 177 F F F 1 020-039 102 102 102 177 177 177 F F F 2 040-059 100 100 100 175 175 175 F F F 3 060-079 99 99 99 174 174 174 F F F 4 080-099 98 98 98 172 172 172 F F F 5 100-119 97 97 97 169 169 169 F F F 6 120-139 99 99 99 174 174 174 F F F 7 140-159 98 98 98 173 173 173 F F F 8 160-179 103 103 103 180 180 180 F F F 9 180-199 90 90 90 158 158 158 F F F 10 200-219 103 103 103 180 180 180 F F F 11 220-239 103 103 103 180 180 180 F F F 12 240-259 94 94 94 166 166 166 F F F 13 260-279 102 102 102 177 177 177 F F F 14 280-299 100 100 100 175 175 175 F F F 15 300-319 103 103 103 180 180 180 F F F 16 320-339 100 100 100 175 175 175 F F F 17 340-359 98 98 98 172 172 172 F F F 18 360-379 98 98 98 170 170 170 F F F 19 380-399 102 102 102 177 177 177 F F F 20 400-419 102 102 102 178 178 178 F F F 21 420-439 102 102 102 178 178 178 F F F 22 440-459 102 102 102 178 178 178 F F F 23 460-479 100 100 100 175 175 175 F F F 24 480-499 94 94 94 165 165 165 F F F 25 500-519 102 102 102 179 179 179 F F F 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 102 102 102 177 177 177 F F F 29 580-599 102 102 102 177 177 177 F F F Max gen: 103, Max flush: 180 Begin loop 52 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.948474235Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.956711842Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957036201Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957063016Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957071305Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957082683Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549} {"msg":"Crucible 3588ad64-4df7-40e7-9bba-5c01ebf3add7 has session id: 18f6f680-c618-4137-a24b-7a48cb5d4fab","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957430698Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"Upstairs opts: Upstairs UUID: 3588ad64-4df7-40e7-9bba-5c01ebf3add7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957460452Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"Crucible stats registered with UUID: 3588ad64-4df7-40e7-9bba-5c01ebf3add7","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.95747161Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:59.957481148Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957616273Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"io task","client":"1","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.95763317Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"io task","client":"0","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957643238Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"io task","client":"2","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957806057Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"task":"crutest"} {"msg":"The guest has requested activation with gen:104","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957823884Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549} {"msg":"3588ad64-4df7-40e7-9bba-5c01ebf3add7 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957859077Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957868436Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"client":"0","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957878094Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"client":"1","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957887762Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"client":"2","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.95789736Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"io task","client":"2","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957907458Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"io task","client":"0","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.957917366Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"io task","client":"1","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"ds_connection connected from Ok(127.0.0.1:38664)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.958054371Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"io task","client":"2","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"ds_connection connected from Ok(127.0.0.1:60407)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.958169519Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"io task","client":"0","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"ds_connection connected from Ok(127.0.0.1:49184)","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.958182097Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"io task","client":"1","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.958755199Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"client":"1","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.958766687Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"client":"0","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.958824467Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"client":"2","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959134129Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959143827Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"[0]R flush_numbers[0..12]: [177, 177, 175, 174, 172, 169, 174, 173, 180, 158, 180, 180]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959168242Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"downstairs","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"[0]R generation[0..12]: [102, 102, 100, 99, 98, 97, 99, 98, 103, 90, 103, 103]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.95917937Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"downstairs","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959190498Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"downstairs","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"[1]R flush_numbers[0..12]: [177, 177, 175, 174, 172, 169, 174, 173, 180, 158, 180, 180]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959201066Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"downstairs","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"[1]R generation[0..12]: [102, 102, 100, 99, 98, 97, 99, 98, 103, 90, 103, 103]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959211414Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"downstairs","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959222212Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"downstairs","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"[2]R flush_numbers[0..12]: [177, 177, 175, 174, 172, 169, 174, 173, 180, 158, 180, 180]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.95923268Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"downstairs","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"[2]R generation[0..12]: [102, 102, 100, 99, 98, 97, 99, 98, 103, 90, 103, 103]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959243008Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"downstairs","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959253866Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"downstairs","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"Max found gen is 104","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959263294Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"downstairs","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"Generation requested: 104 >= found:104","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959272883Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"downstairs","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"Next flush: 181","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959282231Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"downstairs","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959291489Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"":"downstairs","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959303027Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959311745Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"client":"0","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959321253Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"client":"1","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959330892Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"client":"2","":"downstairs","session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.95934073Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959349808Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"3588ad64-4df7-40e7-9bba-5c01ebf3add7 is now active with session: 18f6f680-c618-4137-a24b-7a48cb5d4fab","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959359746Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959368485Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"session_id":"18f6f680-c618-4137-a24b-7a48cb5d4fab"} {"msg":"The guest has finished waiting for activation with:104","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959377643Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549} {"msg":"Activated sub_volume 3588ad64-4df7-40e7-9bba-5c01ebf3add7","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959385681Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.95939373Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.959693154Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"task":"crutest"} 01/30 Write block 318 len 32768 data: 8 8 10 10 6 6 8 8 02/30 Read block 566 len 28672 03/30 Read block 368 len 12288 04/30 Write block 113 len 4096 data: 12 05/30 Write block 28 len 8192 data: 9 11 06/30 Read block 305 len 4096 07/30 Write block 197 len 20480 data: 9 8 7 6 5 08/30 Write block 311 len 4096 data: 11 09/30 Read block 21 len 28672 10/30 Write block 200 len 16384 data: 7 6 7 6 11/30 Write block 105 len 40960 data: 8 8 11 10 13 15 13 13 13 11 12/30 Read block 287 len 24576 13/30 Read block 80 len 4096 14/30 Flush 15/30 Write block 257 len 36864 data: 9 11 10 12 13 12 11 8 6 16/30 Read block 443 len 8192 17/30 Write block 43 len 24576 data: 10 9 10 8 6 7 18/30 Read block 342 len 24576 19/30 Write block 359 len 8192 data: 5 5 20/30 Read block 221 len 28672 21/30 Read block 595 len 12288 22/30 Read block 408 len 36864 23/30 Write block 200 len 24576 data: 8 7 8 7 6 6 24/30 Flush 25/30 Write block 11 len 24576 data: 15 15 13 12 14 15 26/30 Read block 360 len 40960 27/30 Write block 11 len 40960 data: 16 16 14 13 15 16 11 13 12 11 28/30 Write block 447 len 16384 data: 5 3 5 7 29/30 Write block 289 len 40960 data: 10 10 8 9 8 6 4 5 4 5 30/30 Write block 45 len 4096 data: 11 ---------------------------------------------------------------- Crucible gen:104 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Write 2 Done Done Done false 1005 Acked Read 1 Done Done Done false 1006 Acked Write 5 Done Done Sent false 1007 Acked Write 1 Done Done Sent false 1008 Acked Read 7 Done Done Sent false 1009 Acked Write 4 Done Done Sent false 1010 Acked Write 10 Done Done Sent false 1011 Acked Read 6 Done Done Sent false 1012 Acked Read 1 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 9 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Write 6 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Write 2 Done Done Sent false 1019 Acked Read 7 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Write 6 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Read 10 Done Done Sent false 1026 Acked Write 10 Sent Sent Sent false 1027 Acked Write 4 Sent Sent Sent false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 4 4 24 32 Done 26 26 6 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T22:59:59.968379917Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T22:59:59.968581929Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1549,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 102 102 102 177 177 177 T T F <--- 1 020-039 104 104 102 181 181 177 T T T <--- 2 040-059 104 104 100 182 182 175 T T F <--- 3 060-079 99 99 99 174 174 174 F F F 4 080-099 98 98 98 172 172 172 F F F 5 100-119 104 104 97 181 181 169 F F T <--- 6 120-139 99 99 99 174 174 174 F F F 7 140-159 98 98 98 173 173 173 F F F 8 160-179 103 103 103 180 180 180 F F F 9 180-199 104 104 90 181 181 158 F F F <--- 10 200-219 104 104 103 182 182 180 F F F <--- 11 220-239 103 103 103 180 180 180 F F F 12 240-259 104 104 94 182 182 166 F F F <--- 13 260-279 104 104 102 182 182 177 F F F <--- 14 280-299 100 100 100 175 175 175 T T F <--- 15 300-319 104 104 103 181 181 180 F F T <--- 16 320-339 104 104 100 181 181 175 F F T <--- 17 340-359 104 104 98 182 182 172 F F F <--- 18 360-379 104 104 98 182 182 170 F F F <--- 19 380-399 102 102 102 177 177 177 F F F 20 400-419 102 102 102 178 178 178 F F F 21 420-439 102 102 102 178 178 178 F F F 22 440-459 102 102 102 178 178 178 T T F <--- 23 460-479 100 100 100 175 175 175 F F F 24 480-499 94 94 94 165 165 165 F F F 25 500-519 102 102 102 179 179 179 F F F 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 102 102 102 177 177 177 F F F 29 580-599 102 102 102 177 177 177 F F F Max gen: 104, Max flush: 182 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 52 Verifying data now No change after verify Loop: 52 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 105 105 105 183 183 183 F F F 1 020-039 105 105 105 183 183 183 F F F 2 040-059 105 105 105 183 183 183 F F F 3 060-079 99 99 99 174 174 174 F F F 4 080-099 98 98 98 172 172 172 F F F 5 100-119 104 104 104 181 181 181 F F F 6 120-139 99 99 99 174 174 174 F F F 7 140-159 98 98 98 173 173 173 F F F 8 160-179 103 103 103 180 180 180 F F F 9 180-199 104 104 104 181 181 181 F F F 10 200-219 104 104 104 182 182 182 F F F 11 220-239 103 103 103 180 180 180 F F F 12 240-259 104 104 104 182 182 182 F F F 13 260-279 104 104 104 182 182 182 F F F 14 280-299 105 105 105 183 183 183 F F F 15 300-319 104 104 104 181 181 181 F F F 16 320-339 104 104 104 181 181 181 F F F 17 340-359 104 104 104 182 182 182 F F F 18 360-379 104 104 104 182 182 182 F F F 19 380-399 102 102 102 177 177 177 F F F 20 400-419 102 102 102 178 178 178 F F F 21 420-439 102 102 102 178 178 178 F F F 22 440-459 105 105 105 183 183 183 F F F 23 460-479 100 100 100 175 175 175 F F F 24 480-499 94 94 94 165 165 165 F F F 25 500-519 102 102 102 179 179 179 F F F 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 102 102 102 177 177 177 F F F 29 580-599 102 102 102 177 177 177 F F F Max gen: 105, Max flush: 183 Begin loop 53 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.224485135Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.23268137Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233016747Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233045182Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.23305605Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233067778Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557} {"msg":"Crucible cc3cccad-85ba-408e-ae6f-3adc0ee944d1 has session id: 54fa9c95-502d-4ba3-9c25-2a53bfb24cc2","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233439418Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"Upstairs opts: Upstairs UUID: cc3cccad-85ba-408e-ae6f-3adc0ee944d1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233469023Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"Crucible stats registered with UUID: cc3cccad-85ba-408e-ae6f-3adc0ee944d1","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233480221Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:04.233489449Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233634452Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"io task","client":"1","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233665546Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"io task","client":"0","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233676624Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"io task","client":"2","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233818307Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"task":"crutest"} {"msg":"The guest has requested activation with gen:106","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233842543Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557} {"msg":"cc3cccad-85ba-408e-ae6f-3adc0ee944d1 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.23385604Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233864969Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"client":"0","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233874837Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"client":"1","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233886464Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"client":"2","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233896183Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"io task","client":"2","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233906241Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"io task","client":"0","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233916219Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"io task","client":"1","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"ds_connection connected from Ok(127.0.0.1:34220)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.233947713Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"io task","client":"0","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"ds_connection connected from Ok(127.0.0.1:41273)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.234001453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"io task","client":"1","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"ds_connection connected from Ok(127.0.0.1:53050)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.234076229Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"io task","client":"2","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.234588783Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"client":"0","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.234640333Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"client":"1","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.234777568Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"client":"2","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.234889507Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.234908273Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"[0]R flush_numbers[0..12]: [183, 183, 183, 174, 172, 181, 174, 173, 180, 181, 182, 180]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235038859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"downstairs","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"[0]R generation[0..12]: [105, 105, 105, 99, 98, 104, 99, 98, 103, 104, 104, 103]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235051086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"downstairs","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235062144Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"downstairs","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"[1]R flush_numbers[0..12]: [183, 183, 183, 174, 172, 181, 174, 173, 180, 181, 182, 180]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235075582Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"downstairs","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"[1]R generation[0..12]: [105, 105, 105, 99, 98, 104, 99, 98, 103, 104, 104, 103]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.23508614Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"downstairs","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235097308Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"downstairs","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"[2]R flush_numbers[0..12]: [183, 183, 183, 174, 172, 181, 174, 173, 180, 181, 182, 180]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235107886Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"downstairs","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"[2]R generation[0..12]: [105, 105, 105, 99, 98, 104, 99, 98, 103, 104, 104, 103]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235118314Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"downstairs","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235129212Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"downstairs","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"Max found gen is 106","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.23513877Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"downstairs","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"Generation requested: 106 >= found:106","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235148548Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"downstairs","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"Next flush: 184","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235158076Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"downstairs","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235167415Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"":"downstairs","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235177123Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235185941Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"client":"0","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235195599Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"client":"1","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235205267Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"client":"2","":"downstairs","session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235216735Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235225954Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"cc3cccad-85ba-408e-ae6f-3adc0ee944d1 is now active with session: 54fa9c95-502d-4ba3-9c25-2a53bfb24cc2","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235235922Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.23524472Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"session_id":"54fa9c95-502d-4ba3-9c25-2a53bfb24cc2"} {"msg":"The guest has finished waiting for activation with:106","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235253838Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557} {"msg":"Activated sub_volume cc3cccad-85ba-408e-ae6f-3adc0ee944d1","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235261747Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.235269765Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.23561673Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"task":"crutest"} 01/30 Write block 243 len 32768 data: 8 7 9 9 10 10 11 10 02/30 Write block 294 len 24576 data: 7 5 6 5 6 7 03/30 Write block 209 len 12288 data: 10 12 10 04/30 Write block 9 len 20480 data: 8 11 17 17 15 05/30 Write block 230 len 16384 data: 9 7 9 9 06/30 Write block 169 len 24576 data: 4 5 5 7 7 9 07/30 Read block 409 len 20480 08/30 Read block 157 len 12288 09/30 Read block 36 len 40960 10/30 Read block 353 len 24576 11/30 Write block 357 len 16384 data: 9 5 6 6 12/30 Write block 88 len 24576 data: 7 9 7 8 6 8 13/30 Read block 236 len 8192 14/30 Read block 582 len 40960 15/30 Read block 273 len 28672 16/30 Write block 137 len 16384 data: 11 14 16 15 17/30 Flush 18/30 Write block 297 len 20480 data: 6 7 8 8 8 19/30 Write block 1 len 40960 data: 3 3 4 4 5 9 11 9 9 12 20/30 Write block 497 len 16384 data: 6 8 8 9 21/30 Read block 221 len 24576 22/30 Read block 561 len 40960 23/30 Write block 514 len 20480 data: 9 10 10 10 8 24/30 Write block 123 len 32768 data: 11 13 12 14 13 13 14 17 25/30 Read block 261 len 28672 26/30 Read block 222 len 36864 27/30 Write block 109 len 28672 data: 14 16 14 14 14 12 10 28/30 Write block 381 len 16384 data: 10 10 11 10 29/30 Read block 520 len 24576 30/30 Write block 118 len 24576 data: 8 7 8 10 9 12 ---------------------------------------------------------------- Crucible gen:106 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Write 3 Done Done Done false 1003 Acked Write 5 Done Done Done false 1004 Acked Write 4 Done Done Done false 1005 Acked Write 6 Done Done Done false 1006 Acked Read 5 Done Done Done false 1007 Acked Read 3 Done Done Sent false 1008 Acked Read 10 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Write 4 Done Done Sent false 1011 Acked Write 6 Done Done Sent false 1012 Acked Read 2 Done Done Sent false 1013 Acked Read 10 Done Done Sent false 1014 Acked Read 7 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Write 10 Done Done Sent false 1019 Acked Write 4 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Read 10 Done Done Sent false 1022 Acked Write 5 Done Done Sent false 1023 Acked Write 8 Done Done Sent false 1024 Acked Read 7 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Read 6 Done Done Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 23 25 Done 29 29 7 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:04.244151422Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:04.244311662Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1557,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 106 106 105 184 184 183 T T T <--- 1 020-039 105 105 105 183 183 183 F F F 2 040-059 105 105 105 183 183 183 F F F 3 060-079 99 99 99 174 174 174 F F F 4 080-099 106 106 98 184 184 172 F F F <--- 5 100-119 104 104 104 181 181 181 T T F <--- 6 120-139 106 106 99 184 184 174 T T F <--- 7 140-159 106 106 98 184 184 173 F F F <--- 8 160-179 106 106 103 184 184 180 F F T <--- 9 180-199 104 104 104 181 181 181 F F F 10 200-219 106 106 104 184 184 182 F F T <--- 11 220-239 106 106 103 184 184 180 F F T <--- 12 240-259 106 106 104 184 184 182 F F T <--- 13 260-279 104 104 104 182 182 182 F F F 14 280-299 106 106 105 184 184 183 T T T <--- 15 300-319 104 104 104 181 181 181 T T F <--- 16 320-339 104 104 104 181 181 181 F F F 17 340-359 106 106 104 184 184 182 F F F <--- 18 360-379 106 106 104 184 184 182 F F F <--- 19 380-399 102 102 102 177 177 177 T T F <--- 20 400-419 102 102 102 178 178 178 F F F 21 420-439 102 102 102 178 178 178 F F F 22 440-459 105 105 105 183 183 183 F F F 23 460-479 100 100 100 175 175 175 F F F 24 480-499 94 94 94 165 165 165 T T F <--- 25 500-519 102 102 102 179 179 179 T T F <--- 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 102 102 102 177 177 177 F F F 29 580-599 102 102 102 177 177 177 F F F Max gen: 106, Max flush: 184 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 53 Verifying data now No change after verify Loop: 53 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 107 107 107 185 185 185 F F F 1 020-039 105 105 105 183 183 183 F F F 2 040-059 105 105 105 183 183 183 F F F 3 060-079 99 99 99 174 174 174 F F F 4 080-099 106 106 106 184 184 184 F F F 5 100-119 107 107 107 185 185 185 F F F 6 120-139 107 107 107 185 185 185 F F F 7 140-159 106 106 106 184 184 184 F F F 8 160-179 106 106 106 184 184 184 F F F 9 180-199 104 104 104 181 181 181 F F F 10 200-219 106 106 106 184 184 184 F F F 11 220-239 106 106 106 184 184 184 F F F 12 240-259 106 106 106 184 184 184 F F F 13 260-279 104 104 104 182 182 182 F F F 14 280-299 107 107 107 185 185 185 F F F 15 300-319 107 107 107 185 185 185 F F F 16 320-339 104 104 104 181 181 181 F F F 17 340-359 106 106 106 184 184 184 F F F 18 360-379 106 106 106 184 184 184 F F F 19 380-399 107 107 107 185 185 185 F F F 20 400-419 102 102 102 178 178 178 F F F 21 420-439 102 102 102 178 178 178 F F F 22 440-459 105 105 105 183 183 183 F F F 23 460-479 100 100 100 175 175 175 F F F 24 480-499 107 107 107 185 185 185 F F F 25 500-519 107 107 107 185 185 185 F F F 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 102 102 102 177 177 177 F F F 29 580-599 102 102 102 177 177 177 F F F Max gen: 107, Max flush: 185 Begin loop 54 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.505895904Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:08.513009732Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.513050754Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.519938614Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520204285Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520234979Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520243747Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520257005Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565} {"msg":"Crucible 727c399e-a867-4755-b42d-ae32bcea88d3 has session id: aae44c1a-e542-4bfb-a5a7-79cfec81e176","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520565947Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"Upstairs opts: Upstairs UUID: 727c399e-a867-4755-b42d-ae32bcea88d3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520597231Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"Crucible stats registered with UUID: 727c399e-a867-4755-b42d-ae32bcea88d3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520608679Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:08.520617897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520738395Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"io task","client":"2","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520755651Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"io task","client":"0","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.52076559Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"io task","client":"1","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520926959Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"task":"crutest"} {"msg":"The guest has requested activation with gen:108","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520945816Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565} {"msg":"727c399e-a867-4755-b42d-ae32bcea88d3 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520970471Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520979989Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"client":"0","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.520989788Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"client":"1","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.521005505Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"client":"2","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.521015053Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"io task","client":"2","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.521026441Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"io task","client":"1","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.521059285Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"io task","client":"0","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"ds_connection connected from Ok(127.0.0.1:38769)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.52108259Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"io task","client":"0","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"ds_connection connected from Ok(127.0.0.1:58265)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.521093888Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"io task","client":"1","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"ds_connection connected from Ok(127.0.0.1:42449)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.521118373Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"io task","client":"2","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.521716751Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"client":"2","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.521810354Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"client":"1","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.521839858Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"client":"0","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522006217Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522052049Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"[0]R flush_numbers[0..12]: [185, 183, 183, 174, 184, 185, 185, 184, 184, 181, 184, 184]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522138622Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"downstairs","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"[0]R generation[0..12]: [107, 105, 105, 99, 106, 107, 107, 106, 106, 104, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.52214993Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"downstairs","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522161048Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"downstairs","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"[1]R flush_numbers[0..12]: [185, 183, 183, 174, 184, 185, 185, 184, 184, 181, 184, 184]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522171596Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"downstairs","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"[1]R generation[0..12]: [107, 105, 105, 99, 106, 107, 107, 106, 106, 104, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522181984Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"downstairs","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522195122Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"downstairs","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"[2]R flush_numbers[0..12]: [185, 183, 183, 174, 184, 185, 185, 184, 184, 181, 184, 184]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.52220582Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"downstairs","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"[2]R generation[0..12]: [107, 105, 105, 99, 106, 107, 107, 106, 106, 104, 106, 106]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522216328Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"downstairs","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522227066Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"downstairs","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"Max found gen is 108","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522236404Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"downstairs","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"Generation requested: 108 >= found:108","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522246012Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"downstairs","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"Next flush: 186","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522255281Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"downstairs","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522264509Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"":"downstairs","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522274177Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522282875Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"client":"0","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522292584Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"client":"1","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522302082Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"client":"2","":"downstairs","session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.52231187Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522320968Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"727c399e-a867-4755-b42d-ae32bcea88d3 is now active with session: aae44c1a-e542-4bfb-a5a7-79cfec81e176","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522330766Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522340904Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"session_id":"aae44c1a-e542-4bfb-a5a7-79cfec81e176"} {"msg":"The guest has finished waiting for activation with:108","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522350073Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565} {"msg":"Activated sub_volume 727c399e-a867-4755-b42d-ae32bcea88d3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.522357981Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.52236597Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.52268432Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"task":"crutest"} 01/30 Write block 182 len 4096 data: 9 02/30 Read block 19 len 16384 03/30 Write block 240 len 12288 data: 6 8 8 04/30 Read block 123 len 32768 05/30 Read block 221 len 20480 06/30 Read block 113 len 32768 07/30 Read block 251 len 12288 08/30 Read block 591 len 16384 09/30 Read block 243 len 20480 10/30 Read block 531 len 28672 11/30 Write block 374 len 36864 data: 9 12 10 10 10 10 12 11 11 12/30 Flush 13/30 Write block 332 len 12288 data: 12 10 10 14/30 Write block 301 len 28672 data: 9 12 14 16 13 13 14 15/30 Read block 434 len 36864 16/30 Read block 499 len 32768 17/30 Read block 316 len 8192 18/30 Write block 397 len 12288 data: 12 11 9 19/30 Read block 346 len 28672 20/30 Read block 306 len 16384 21/30 Write block 429 len 36864 data: 6 6 3 3 4 5 7 5 6 22/30 Read block 249 len 12288 23/30 Write block 234 len 28672 data: 10 9 10 10 10 5 7 24/30 Flush 25/30 Flush 26/30 Write block 147 len 24576 data: 9 6 5 4 6 7 27/30 Read block 363 len 4096 28/30 Read block 47 len 36864 29/30 Write block 39 len 28672 data: 8 8 9 10 11 10 12 30/30 Write block 190 len 20480 data: 7 11 13 15 12 ---------------------------------------------------------------- Crucible gen:108 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Sent Done Done false 1001 Acked Read 4 Sent Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Read 8 Sent Done Done false 1004 Acked Read 5 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Read 3 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Read 5 Sent Done Done false 1009 Acked Read 7 Sent Done Done false 1010 Acked Write 9 Sent Done Done false 1011 Acked Flush 0 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Write 7 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Read 8 Sent Done Done false 1016 Acked Read 2 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Read 7 Sent Done Done false 1019 Acked Read 4 Sent Done Done false 1020 Acked Write 9 Sent Done Done false 1021 Acked Read 3 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Flush 0 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Read 9 Sent Done Done false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 2 2 34 Done 0 28 28 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1024)) Some(JobId(1024)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:08.531963212Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:08.53219128Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1565,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 107 107 107 185 185 185 F F F 1 020-039 105 105 105 183 183 183 F T T <--- 2 040-059 105 105 105 183 183 183 F T T <--- 3 060-079 99 99 99 174 174 174 F F F 4 080-099 106 106 106 184 184 184 F F F 5 100-119 107 107 107 185 185 185 F F F 6 120-139 107 107 107 185 185 185 F F F 7 140-159 106 106 106 184 184 184 F T T <--- 8 160-179 106 106 106 184 184 184 F F F 9 180-199 104 108 108 181 186 186 F T T <--- 10 200-219 106 106 106 184 184 184 F F F 11 220-239 106 108 108 184 187 187 F F F <--- 12 240-259 106 108 108 184 187 187 F F F <--- 13 260-279 104 104 104 182 182 182 F F F 14 280-299 107 107 107 185 185 185 F F F 15 300-319 107 108 108 185 187 187 F F F <--- 16 320-339 104 108 108 181 187 187 F F F <--- 17 340-359 106 106 106 184 184 184 F F F 18 360-379 106 108 108 184 186 186 F F F <--- 19 380-399 107 108 108 185 187 187 F F F <--- 20 400-419 102 102 102 178 178 178 F F F 21 420-439 102 108 108 178 187 187 F F F <--- 22 440-459 105 105 105 183 183 183 F F F 23 460-479 100 100 100 175 175 175 F F F 24 480-499 107 107 107 185 185 185 F F F 25 500-519 107 107 107 185 185 185 F F F 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 102 102 102 177 177 177 F F F 29 580-599 102 102 102 177 177 177 F F F Max gen: 108, Max flush: 187 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 54 Verifying data now No change after verify Loop: 54 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 107 107 107 185 185 185 F F F 1 020-039 109 109 109 188 188 188 F F F 2 040-059 109 109 109 188 188 188 F F F 3 060-079 99 99 99 174 174 174 F F F 4 080-099 106 106 106 184 184 184 F F F 5 100-119 107 107 107 185 185 185 F F F 6 120-139 107 107 107 185 185 185 F F F 7 140-159 109 109 109 188 188 188 F F F 8 160-179 106 106 106 184 184 184 F F F 9 180-199 109 109 109 188 188 188 F F F 10 200-219 106 106 106 184 184 184 F F F 11 220-239 108 108 108 187 187 187 F F F 12 240-259 108 108 108 187 187 187 F F F 13 260-279 104 104 104 182 182 182 F F F 14 280-299 107 107 107 185 185 185 F F F 15 300-319 108 108 108 187 187 187 F F F 16 320-339 108 108 108 187 187 187 F F F 17 340-359 106 106 106 184 184 184 F F F 18 360-379 108 108 108 186 186 186 F F F 19 380-399 108 108 108 187 187 187 F F F 20 400-419 102 102 102 178 178 178 F F F 21 420-439 108 108 108 187 187 187 F F F 22 440-459 105 105 105 183 183 183 F F F 23 460-479 100 100 100 175 175 175 F F F 24 480-499 107 107 107 185 185 185 F F F 25 500-519 107 107 107 185 185 185 F F F 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 102 102 102 177 177 177 F F F 29 580-599 102 102 102 177 177 177 F F F Max gen: 109, Max flush: 188 Begin loop 55 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.785154756Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:12.79228723Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.79234321Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.799108123Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.799376273Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.799404697Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.799413326Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.799424644Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573} {"msg":"Crucible 6c2a85f4-8fa4-4ea6-8f18-d375656a0170 has session id: 6ab49952-0f33-410f-bb19-40e62bf5d607","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.799746093Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"Upstairs opts: Upstairs UUID: 6c2a85f4-8fa4-4ea6-8f18-d375656a0170, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.799776628Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"Crucible stats registered with UUID: 6c2a85f4-8fa4-4ea6-8f18-d375656a0170","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.799790125Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:12.799799203Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.799943736Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"io task","client":"1","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.799962573Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"io task","client":"2","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.799972511Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"io task","client":"0","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.800115044Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"task":"crutest"} {"msg":"The guest has requested activation with gen:110","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.80013945Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573} {"msg":"6c2a85f4-8fa4-4ea6-8f18-d375656a0170 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.800148568Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.800157756Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"client":"0","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.800167744Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"client":"1","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.800177383Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"client":"2","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.800186691Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"io task","client":"2","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.800196639Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"io task","client":"0","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.800206547Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"io task","client":"1","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"ds_connection connected from Ok(127.0.0.1:55074)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.800262337Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"io task","client":"1","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"ds_connection connected from Ok(127.0.0.1:33259)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.800307578Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"io task","client":"0","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"ds_connection connected from Ok(127.0.0.1:49705)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.800360478Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"io task","client":"2","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.800984981Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"client":"2","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801003208Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"client":"1","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801041321Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"client":"0","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801326577Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801359181Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"[0]R flush_numbers[0..12]: [185, 188, 188, 174, 184, 185, 185, 188, 184, 188, 184, 187]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801416411Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"downstairs","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"[0]R generation[0..12]: [107, 109, 109, 99, 106, 107, 107, 109, 106, 109, 106, 108]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801428258Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"downstairs","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801439286Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"downstairs","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"[1]R flush_numbers[0..12]: [185, 188, 188, 174, 184, 185, 185, 188, 184, 188, 184, 187]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801449914Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"downstairs","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"[1]R generation[0..12]: [107, 109, 109, 99, 106, 107, 107, 109, 106, 109, 106, 108]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801460382Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"downstairs","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.80147121Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"downstairs","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"[2]R flush_numbers[0..12]: [185, 188, 188, 174, 184, 185, 185, 188, 184, 188, 184, 187]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801482018Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"downstairs","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"[2]R generation[0..12]: [107, 109, 109, 99, 106, 107, 107, 109, 106, 109, 106, 108]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801495216Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"downstairs","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801506304Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"downstairs","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"Max found gen is 110","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801515742Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"downstairs","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"Generation requested: 110 >= found:110","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.80152544Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"downstairs","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"Next flush: 189","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801534728Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"downstairs","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801544017Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"":"downstairs","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801553745Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801562453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"client":"0","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801572321Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"client":"1","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801582Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"client":"2","":"downstairs","session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801592088Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801601186Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"6c2a85f4-8fa4-4ea6-8f18-d375656a0170 is now active with session: 6ab49952-0f33-410f-bb19-40e62bf5d607","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801611094Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801619673Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"session_id":"6ab49952-0f33-410f-bb19-40e62bf5d607"} {"msg":"The guest has finished waiting for activation with:110","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801628651Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573} {"msg":"Activated sub_volume 6c2a85f4-8fa4-4ea6-8f18-d375656a0170","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801636609Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801644548Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.801932764Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"task":"crutest"} 01/30 Write block 30 len 12288 data: 14 14 14 02/30 Read block 123 len 36864 03/30 Read block 258 len 16384 04/30 Read block 106 len 4096 05/30 Write block 137 len 24576 data: 12 15 17 16 15 12 06/30 Flush 07/30 Write block 441 len 12288 data: 7 8 8 08/30 Flush 09/30 Write block 182 len 36864 data: 10 9 7 7 8 5 5 7 8 10/30 Read block 328 len 16384 11/30 Read block 231 len 28672 12/30 Read block 29 len 8192 13/30 Read block 326 len 28672 14/30 Write block 240 len 32768 data: 8 9 9 9 8 10 10 11 15/30 Flush 16/30 Write block 281 len 8192 data: 6 5 17/30 Read block 171 len 12288 18/30 Read block 251 len 40960 19/30 Read block 122 len 20480 20/30 Write block 507 len 8192 data: 8 4 21/30 Write block 41 len 12288 data: 10 11 12 22/30 Write block 398 len 4096 data: 12 23/30 Write block 4 len 8192 data: 5 6 24/30 Read block 335 len 12288 25/30 Write block 399 len 40960 data: 10 6 7 9 10 11 8 10 7 6 26/30 Write block 177 len 32768 data: 8 12 10 8 7 11 10 8 27/30 Write block 579 len 16384 data: 12 11 11 11 28/30 Write block 483 len 8192 data: 7 7 29/30 Write block 323 len 40960 data: 7 9 9 9 8 6 8 10 10 13 30/30 Read block 592 len 32768 ---------------------------------------------------------------- Crucible gen:110 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 9 Sent Done Done false 1002 Acked Read 4 Sent Done Done false 1003 Acked Read 1 Sent Done Done false 1004 Acked Write 6 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 3 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 9 Sent Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Read 7 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Read 7 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Read 3 Sent Done Done false 1017 Acked Read 10 Sent Done Done false 1018 Acked Read 5 Sent Done Done false 1019 Acked Write 2 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Write 1 Sent Done Done false 1022 Acked Write 2 Sent Done Done false 1023 Acked Read 3 Sent Done Done false 1024 Acked Write 10 Sent Done Done false 1025 Acked Write 8 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Write 2 Sent Done Done false 1028 Acked Write 10 Sent Done Done false 1029 Acked Read 8 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 1 30 Done 1 30 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1014)) Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:12.810133288Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:12.810323432Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1573,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 107 107 107 185 185 185 F T T <--- 1 020-039 109 110 110 188 189 189 T F F <--- 2 040-059 109 109 109 188 188 188 F T T <--- 3 060-079 99 99 99 174 174 174 F F F 4 080-099 106 106 106 184 184 184 F F F 5 100-119 107 107 107 185 185 185 F F F 6 120-139 107 110 110 185 189 189 F F F <--- 7 140-159 109 110 110 188 189 189 F F F <--- 8 160-179 106 106 106 184 184 184 F T T <--- 9 180-199 109 110 110 188 191 191 F T T <--- 10 200-219 106 106 106 184 184 184 F F F 11 220-239 108 108 108 187 187 187 F F F 12 240-259 108 110 110 187 191 191 F F F <--- 13 260-279 104 104 104 182 182 182 F F F 14 280-299 107 107 107 185 185 185 F T T <--- 15 300-319 108 108 108 187 187 187 F F F 16 320-339 108 108 108 187 187 187 F T T <--- 17 340-359 106 106 106 184 184 184 F F F 18 360-379 108 108 108 186 186 186 F F F 19 380-399 108 108 108 187 187 187 F T T <--- 20 400-419 102 102 102 178 178 178 F T T <--- 21 420-439 108 108 108 187 187 187 F F F 22 440-459 105 110 110 183 190 190 F F F <--- 23 460-479 100 100 100 175 175 175 F F F 24 480-499 107 107 107 185 185 185 F T T <--- 25 500-519 107 107 107 185 185 185 F T T <--- 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 102 102 102 177 177 177 F T T <--- 29 580-599 102 102 102 177 177 177 F T T <--- Max gen: 110, Max flush: 191 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 55 Verifying data now No change after verify Loop: 55 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 192 192 192 F F F 1 020-039 110 110 110 189 189 189 F F F 2 040-059 111 111 111 192 192 192 F F F 3 060-079 99 99 99 174 174 174 F F F 4 080-099 106 106 106 184 184 184 F F F 5 100-119 107 107 107 185 185 185 F F F 6 120-139 110 110 110 189 189 189 F F F 7 140-159 110 110 110 189 189 189 F F F 8 160-179 111 111 111 192 192 192 F F F 9 180-199 111 111 111 192 192 192 F F F 10 200-219 106 106 106 184 184 184 F F F 11 220-239 108 108 108 187 187 187 F F F 12 240-259 110 110 110 191 191 191 F F F 13 260-279 104 104 104 182 182 182 F F F 14 280-299 111 111 111 192 192 192 F F F 15 300-319 108 108 108 187 187 187 F F F 16 320-339 111 111 111 192 192 192 F F F 17 340-359 106 106 106 184 184 184 F F F 18 360-379 108 108 108 186 186 186 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 111 111 111 192 192 192 F F F 21 420-439 108 108 108 187 187 187 F F F 22 440-459 110 110 110 190 190 190 F F F 23 460-479 100 100 100 175 175 175 F F F 24 480-499 111 111 111 192 192 192 F F F 25 500-519 111 111 111 192 192 192 F F F 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 111 111 111 192 192 192 F F F 29 580-599 111 111 111 192 192 192 F F F Max gen: 111, Max flush: 192 Begin loop 56 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.085112931Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.09344757Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.093750363Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.093777488Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.093786147Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.093797575Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581} {"msg":"Crucible 99153171-f091-4dbb-884e-7a07323357c3 has session id: 572603fa-9bbc-4eff-9a25-c6ca61498b4f","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094166475Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"Upstairs opts: Upstairs UUID: 99153171-f091-4dbb-884e-7a07323357c3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094189741Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"Crucible stats registered with UUID: 99153171-f091-4dbb-884e-7a07323357c3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094199479Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:17.094209487Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094375616Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"io task","client":"1","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094405081Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"io task","client":"0","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094418128Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"io task","client":"2","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.09451453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"task":"crutest"} {"msg":"The guest has requested activation with gen:112","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094541845Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581} {"msg":"99153171-f091-4dbb-884e-7a07323357c3 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094554853Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094563651Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"client":"0","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094573159Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"client":"1","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094582698Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"client":"2","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094592096Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"io task","client":"2","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094602004Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"io task","client":"0","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094612062Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"io task","client":"1","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"ds_connection connected from Ok(127.0.0.1:40257)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094640867Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"io task","client":"2","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"ds_connection connected from Ok(127.0.0.1:32810)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094705994Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"io task","client":"1","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"ds_connection connected from Ok(127.0.0.1:60362)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.094818433Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"io task","client":"0","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095389796Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"client":"0","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095408503Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"client":"1","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095419871Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"client":"2","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095696519Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095734442Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"[0]R flush_numbers[0..12]: [192, 189, 192, 174, 184, 185, 189, 189, 192, 192, 184, 187]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095787902Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"downstairs","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"[0]R generation[0..12]: [111, 110, 111, 99, 106, 107, 110, 110, 111, 111, 106, 108]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.09579941Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"downstairs","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095810828Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"downstairs","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"[1]R flush_numbers[0..12]: [192, 189, 192, 174, 184, 185, 189, 189, 192, 192, 184, 187]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095821486Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"downstairs","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"[1]R generation[0..12]: [111, 110, 111, 99, 106, 107, 110, 110, 111, 111, 106, 108]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095832274Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"downstairs","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095843161Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"downstairs","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"[2]R flush_numbers[0..12]: [192, 189, 192, 174, 184, 185, 189, 189, 192, 192, 184, 187]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095853739Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"downstairs","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"[2]R generation[0..12]: [111, 110, 111, 99, 106, 107, 110, 110, 111, 111, 106, 108]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095864447Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"downstairs","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095875405Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"downstairs","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"Max found gen is 112","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095884984Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"downstairs","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"Generation requested: 112 >= found:112","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095894892Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"downstairs","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"Next flush: 193","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.09590426Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"downstairs","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095916328Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"":"downstairs","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095926456Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095935354Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"client":"0","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095945372Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"client":"1","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.09595534Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"client":"2","":"downstairs","session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095965199Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095974367Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"99153171-f091-4dbb-884e-7a07323357c3 is now active with session: 572603fa-9bbc-4eff-9a25-c6ca61498b4f","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095984355Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.095993143Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"session_id":"572603fa-9bbc-4eff-9a25-c6ca61498b4f"} {"msg":"The guest has finished waiting for activation with:112","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.096002252Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581} {"msg":"Activated sub_volume 99153171-f091-4dbb-884e-7a07323357c3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.0960103Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.096018739Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.096379121Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"task":"crutest"} 01/30 Write block 230 len 20480 data: 10 8 10 10 11 02/30 Read block 18 len 8192 03/30 Write block 88 len 12288 data: 8 10 8 04/30 Read block 26 len 40960 05/30 Flush 06/30 Write block 38 len 28672 data: 9 9 9 11 12 13 11 07/30 Write block 127 len 8192 data: 14 14 08/30 Read block 246 len 12288 09/30 Read block 381 len 8192 10/30 Read block 331 len 40960 11/30 Read block 87 len 16384 12/30 Read block 500 len 24576 13/30 Read block 459 len 40960 14/30 Write block 198 len 4096 data: 9 15/30 Read block 102 len 4096 16/30 Read block 72 len 8192 17/30 Flush 18/30 Write block 46 len 8192 data: 9 7 19/30 Read block 241 len 36864 20/30 Write block 565 len 12288 data: 10 9 8 21/30 Read block 404 len 8192 22/30 Read block 59 len 36864 23/30 Read block 521 len 20480 24/30 Read block 326 len 28672 25/30 Write block 193 len 40960 data: 16 13 13 13 10 10 8 9 8 9 26/30 Read block 566 len 36864 27/30 Read block 415 len 24576 28/30 Read block 212 len 24576 29/30 Write block 291 len 40960 data: 9 10 9 8 6 7 7 8 9 9 30/30 Write block 42 len 24576 data: 13 14 12 13 10 8 ---------------------------------------------------------------- Crucible gen:112 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Sent false 1001 Acked Read 2 Done Done Sent false 1002 Acked Write 3 Done Done Sent false 1003 Acked Read 10 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Write 2 Done Done Sent false 1007 Acked Read 3 Done Done Sent false 1008 Acked Read 2 Done Done Sent false 1009 Acked Read 10 Done Done Sent false 1010 Acked Read 4 Done Done Sent false 1011 Acked Read 6 Done Done Sent false 1012 Acked Read 10 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Read 1 Done Done Sent false 1015 Acked Read 2 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 2 Done Done Sent false 1018 Acked Read 9 Done Done Sent false 1019 Acked Write 3 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Read 9 Done Done Sent false 1022 Acked Read 5 Done Done Sent false 1023 Acked Read 7 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Read 9 Done Done Sent false 1026 Acked Read 6 Done Done Sent false 1027 Acked Read 6 Done Done Sent false 1028 Acked Write 10 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 30 34 Done 28 28 0 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:17.105124213Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:17.105282354Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1581,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 192 192 192 F F F 1 020-039 112 112 110 194 194 189 F F F <--- 2 040-059 112 112 111 194 194 192 T T F <--- 3 060-079 99 99 99 174 174 174 F F F 4 080-099 112 112 106 193 193 184 F F F <--- 5 100-119 107 107 107 185 185 185 F F F 6 120-139 112 112 110 194 194 189 F F F <--- 7 140-159 110 110 110 189 189 189 F F F 8 160-179 111 111 111 192 192 192 F F F 9 180-199 112 112 111 194 194 192 T T F <--- 10 200-219 106 106 106 184 184 184 T T F <--- 11 220-239 112 112 108 193 193 187 F F F <--- 12 240-259 110 110 110 191 191 191 F F F 13 260-279 104 104 104 182 182 182 F F F 14 280-299 111 111 111 192 192 192 T T F <--- 15 300-319 108 108 108 187 187 187 T T F <--- 16 320-339 111 111 111 192 192 192 F F F 17 340-359 106 106 106 184 184 184 F F F 18 360-379 108 108 108 186 186 186 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 111 111 111 192 192 192 F F F 21 420-439 108 108 108 187 187 187 F F F 22 440-459 110 110 110 190 190 190 F F F 23 460-479 100 100 100 175 175 175 F F F 24 480-499 111 111 111 192 192 192 F F F 25 500-519 111 111 111 192 192 192 F F F 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 111 111 111 192 192 192 T T F <--- 29 580-599 111 111 111 192 192 192 F F F Max gen: 112, Max flush: 194 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 56 Verifying data now No change after verify Loop: 56 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 192 192 192 F F F 1 020-039 112 112 112 194 194 194 F F F 2 040-059 113 113 113 195 195 195 F F F 3 060-079 99 99 99 174 174 174 F F F 4 080-099 112 112 112 193 193 193 F F F 5 100-119 107 107 107 185 185 185 F F F 6 120-139 112 112 112 194 194 194 F F F 7 140-159 110 110 110 189 189 189 F F F 8 160-179 111 111 111 192 192 192 F F F 9 180-199 113 113 113 195 195 195 F F F 10 200-219 113 113 113 195 195 195 F F F 11 220-239 112 112 112 193 193 193 F F F 12 240-259 110 110 110 191 191 191 F F F 13 260-279 104 104 104 182 182 182 F F F 14 280-299 113 113 113 195 195 195 F F F 15 300-319 113 113 113 195 195 195 F F F 16 320-339 111 111 111 192 192 192 F F F 17 340-359 106 106 106 184 184 184 F F F 18 360-379 108 108 108 186 186 186 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 111 111 111 192 192 192 F F F 21 420-439 108 108 108 187 187 187 F F F 22 440-459 110 110 110 190 190 190 F F F 23 460-479 100 100 100 175 175 175 F F F 24 480-499 111 111 111 192 192 192 F F F 25 500-519 111 111 111 192 192 192 F F F 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 113 113 113 195 195 195 F F F 29 580-599 111 111 111 192 192 192 F F F Max gen: 113, Max flush: 195 Begin loop 57 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.353353137Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:21.360561946Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.360618146Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.367365482Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.367640211Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.367673075Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.367681503Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.367692891Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589} {"msg":"Crucible 8db4cade-b460-425a-84ee-fa45827f52f5 has session id: cd8a9d07-3994-4ffd-b483-993f21a629a2","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.367979497Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"Upstairs opts: Upstairs UUID: 8db4cade-b460-425a-84ee-fa45827f52f5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368025739Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"Crucible stats registered with UUID: 8db4cade-b460-425a-84ee-fa45827f52f5","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368036157Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:21.368045285Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.3681775Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"io task","client":"1","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368194457Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"io task","client":"0","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368204475Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"io task","client":"2","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368372184Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"task":"crutest"} {"msg":"The guest has requested activation with gen:114","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368388751Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589} {"msg":"8db4cade-b460-425a-84ee-fa45827f52f5 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368420515Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368431833Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"client":"0","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368441611Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"client":"1","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368451099Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"client":"2","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368460557Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"io task","client":"2","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368470545Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"io task","client":"0","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368480583Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"io task","client":"1","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"ds_connection connected from Ok(127.0.0.1:35475)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368531434Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"io task","client":"1","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"ds_connection connected from Ok(127.0.0.1:61123)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368542442Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"io task","client":"2","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"ds_connection connected from Ok(127.0.0.1:44868)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.368638874Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"io task","client":"0","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369323916Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"client":"2","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369341742Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"client":"1","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369361089Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"client":"0","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369638097Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.36967341Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"[0]R flush_numbers[0..12]: [192, 194, 195, 174, 193, 185, 194, 189, 192, 195, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369723591Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"downstairs","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"[0]R generation[0..12]: [111, 112, 113, 99, 112, 107, 112, 110, 111, 113, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369737508Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"downstairs","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369748426Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"downstairs","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"[1]R flush_numbers[0..12]: [192, 194, 195, 174, 193, 185, 194, 189, 192, 195, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369758954Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"downstairs","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"[1]R generation[0..12]: [111, 112, 113, 99, 112, 107, 112, 110, 111, 113, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369769262Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"downstairs","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.36977991Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"downstairs","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"[2]R flush_numbers[0..12]: [192, 194, 195, 174, 193, 185, 194, 189, 192, 195, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369790398Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"downstairs","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"[2]R generation[0..12]: [111, 112, 113, 99, 112, 107, 112, 110, 111, 113, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369800816Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"downstairs","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369811624Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"downstairs","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"Max found gen is 114","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369820892Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"downstairs","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"Generation requested: 114 >= found:114","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369830441Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"downstairs","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"Next flush: 196","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369839719Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"downstairs","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369849027Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"":"downstairs","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369858725Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369867314Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"client":"0","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369877842Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"client":"1","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.36988736Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"client":"2","":"downstairs","session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369897088Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369906086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"8db4cade-b460-425a-84ee-fa45827f52f5 is now active with session: cd8a9d07-3994-4ffd-b483-993f21a629a2","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369915985Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369924663Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"session_id":"cd8a9d07-3994-4ffd-b483-993f21a629a2"} {"msg":"The guest has finished waiting for activation with:114","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369933651Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589} {"msg":"Activated sub_volume 8db4cade-b460-425a-84ee-fa45827f52f5","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.36994161Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.369949608Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.370284996Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"task":"crutest"} 01/30 Write block 357 len 40960 data: 10 6 7 7 2 3 3 7 8 6 02/30 Write block 258 len 16384 data: 12 11 13 14 03/30 Flush 04/30 Write block 194 len 16384 data: 14 14 14 11 05/30 Read block 82 len 36864 06/30 Flush 07/30 Write block 99 len 20480 data: 4 7 6 6 8 08/30 Write block 359 len 4096 data: 8 09/30 Read block 393 len 4096 10/30 Write block 64 len 24576 data: 9 9 10 10 10 7 11/30 Write block 109 len 12288 data: 15 17 15 12/30 Write block 350 len 20480 data: 7 9 8 8 8 13/30 Flush 14/30 Write block 581 len 40960 data: 12 12 8 6 5 7 10 11 12 13 15/30 Read block 45 len 40960 16/30 Write block 307 len 20480 data: 15 12 11 12 12 17/30 Write block 172 len 28672 data: 8 8 10 7 7 9 13 18/30 Read block 574 len 12288 19/30 Read block 181 len 16384 20/30 Read block 328 len 20480 21/30 Write block 424 len 32768 data: 8 6 6 7 6 7 7 4 22/30 Write block 114 len 24576 data: 13 11 7 6 9 8 23/30 Read block 442 len 40960 24/30 Read block 511 len 4096 25/30 Read block 506 len 12288 26/30 Write block 444 len 4096 data: 7 27/30 Read block 177 len 12288 28/30 Write block 116 len 24576 data: 8 7 10 9 9 11 29/30 Read block 549 len 20480 30/30 Read block 308 len 8192 ---------------------------------------------------------------- Crucible gen:114 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 4 Done Done Done false 1004 Acked Read 9 Done Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Read 1 Sent Done Done false 1009 Acked Write 6 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Write 5 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Read 10 Sent Done Done false 1015 Acked Write 5 Sent Done Done false 1016 Acked Write 7 Sent Done Done false 1017 Acked Read 3 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Write 6 Sent Done Done false 1022 Acked Read 10 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Read 3 Sent Done Done false 1027 Acked Write 6 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Read 2 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 25 0 0 25 Done 2 27 27 56 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1012)) Some(JobId(1012)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:21.379139797Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:21.379315084Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1589,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 192 192 192 F F F 1 020-039 112 112 112 194 194 194 F F F 2 040-059 113 113 113 195 195 195 F F F 3 060-079 99 114 114 174 198 198 F F F <--- 4 080-099 112 114 114 193 198 198 F F F <--- 5 100-119 107 114 114 185 198 198 F T T <--- 6 120-139 112 112 112 194 194 194 F T T <--- 7 140-159 110 110 110 189 189 189 F F F 8 160-179 111 111 111 192 192 192 F T T <--- 9 180-199 113 114 114 195 197 197 T F F <--- 10 200-219 113 113 113 195 195 195 F F F 11 220-239 112 112 112 193 193 193 F F F 12 240-259 114 114 114 196 196 196 F F F 13 260-279 114 114 114 196 196 196 F F F 14 280-299 113 113 113 195 195 195 F F F 15 300-319 113 113 113 195 195 195 F T T <--- 16 320-339 111 111 111 192 192 192 F F F 17 340-359 114 114 114 196 198 198 F F F <--- 18 360-379 114 114 114 196 196 196 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 111 111 111 192 192 192 F F F 21 420-439 108 108 108 187 187 187 F T T <--- 22 440-459 110 110 110 190 190 190 F T T <--- 23 460-479 100 100 100 175 175 175 F F F 24 480-499 111 111 111 192 192 192 F F F 25 500-519 111 111 111 192 192 192 F F F 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 113 113 113 195 195 195 F F F 29 580-599 111 111 111 192 192 192 F T T <--- Max gen: 114, Max flush: 198 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 57 Verifying data now No change after verify Loop: 57 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 192 192 192 F F F 1 020-039 112 112 112 194 194 194 F F F 2 040-059 113 113 113 195 195 195 F F F 3 060-079 114 114 114 198 198 198 F F F 4 080-099 114 114 114 198 198 198 F F F 5 100-119 115 115 115 199 199 199 F F F 6 120-139 115 115 115 199 199 199 F F F 7 140-159 110 110 110 189 189 189 F F F 8 160-179 115 115 115 199 199 199 F F F 9 180-199 114 114 114 197 197 197 F F F 10 200-219 113 113 113 195 195 195 F F F 11 220-239 112 112 112 193 193 193 F F F 12 240-259 114 114 114 196 196 196 F F F 13 260-279 114 114 114 196 196 196 F F F 14 280-299 113 113 113 195 195 195 F F F 15 300-319 115 115 115 199 199 199 F F F 16 320-339 111 111 111 192 192 192 F F F 17 340-359 114 114 114 198 198 198 F F F 18 360-379 114 114 114 196 196 196 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 111 111 111 192 192 192 F F F 21 420-439 115 115 115 199 199 199 F F F 22 440-459 115 115 115 199 199 199 F F F 23 460-479 100 100 100 175 175 175 F F F 24 480-499 111 111 111 192 192 192 F F F 25 500-519 111 111 111 192 192 192 F F F 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 113 113 113 195 195 195 F F F 29 580-599 115 115 115 199 199 199 F F F Max gen: 115, Max flush: 199 Begin loop 58 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.636519047Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.644768142Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645069495Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.64509792Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645106589Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645118096Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597} {"msg":"Crucible 536ba1db-b7c0-461d-b3a1-d02953696c47 has session id: be09e857-c3fe-4cc7-bea2-f39bafba9123","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645396414Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"Upstairs opts: Upstairs UUID: 536ba1db-b7c0-461d-b3a1-d02953696c47, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.64547486Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"Crucible stats registered with UUID: 536ba1db-b7c0-461d-b3a1-d02953696c47","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645500875Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:25.645511173Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645641718Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"io task","client":"1","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645659375Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"io task","client":"0","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645669453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"io task","client":"2","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645788221Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"task":"crutest"} {"msg":"The guest has requested activation with gen:116","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645803948Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597} {"msg":"536ba1db-b7c0-461d-b3a1-d02953696c47 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645821155Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645829823Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"client":"0","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645839431Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"client":"1","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645849019Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"client":"2","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645858578Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"io task","client":"2","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645868426Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"io task","client":"0","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645880304Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"io task","client":"1","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"ds_connection connected from Ok(127.0.0.1:40149)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645962968Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"io task","client":"0","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"ds_connection connected from Ok(127.0.0.1:37863)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.645998891Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"io task","client":"2","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"ds_connection connected from Ok(127.0.0.1:38779)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.646010259Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"io task","client":"1","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.646776566Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"client":"0","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.646795102Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"client":"1","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.6468058Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"client":"2","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647109643Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647119542Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"[0]R flush_numbers[0..12]: [192, 194, 195, 198, 198, 199, 199, 189, 199, 197, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647154765Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"downstairs","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"[0]R generation[0..12]: [111, 112, 113, 114, 114, 115, 115, 110, 115, 114, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647166163Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"downstairs","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647177161Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"downstairs","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"[1]R flush_numbers[0..12]: [192, 194, 195, 198, 198, 199, 199, 189, 199, 197, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647187749Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"downstairs","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"[1]R generation[0..12]: [111, 112, 113, 114, 114, 115, 115, 110, 115, 114, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647198177Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"downstairs","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647211344Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"downstairs","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"[2]R flush_numbers[0..12]: [192, 194, 195, 198, 198, 199, 199, 189, 199, 197, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647221872Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"downstairs","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"[2]R generation[0..12]: [111, 112, 113, 114, 114, 115, 115, 110, 115, 114, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.64723219Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"downstairs","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647243238Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"downstairs","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"Max found gen is 116","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647252557Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"downstairs","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"Generation requested: 116 >= found:116","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647262245Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"downstairs","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"Next flush: 200","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647271383Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"downstairs","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647280711Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"":"downstairs","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.64729034Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647299008Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"client":"0","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647308716Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"client":"1","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647318184Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"client":"2","":"downstairs","session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647327992Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647336901Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"536ba1db-b7c0-461d-b3a1-d02953696c47 is now active with session: be09e857-c3fe-4cc7-bea2-f39bafba9123","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647346969Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647355497Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"session_id":"be09e857-c3fe-4cc7-bea2-f39bafba9123"} {"msg":"The guest has finished waiting for activation with:116","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647365745Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597} {"msg":"Activated sub_volume 536ba1db-b7c0-461d-b3a1-d02953696c47","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647373594Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647381702Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.647708911Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"task":"crutest"} 01/30 Write block 486 len 4096 data: 8 02/30 Read block 458 len 28672 03/30 Read block 284 len 40960 04/30 Write block 164 len 28672 data: 7 8 8 7 5 5 6 05/30 Write block 366 len 36864 data: 7 5 4 5 6 9 9 8 10 06/30 Write block 91 len 36864 data: 9 7 9 9 7 7 5 3 5 07/30 Write block 476 len 40960 data: 6 6 6 6 4 4 8 8 8 7 08/30 Read block 235 len 40960 09/30 Read block 180 len 12288 10/30 Read block 194 len 12288 11/30 Read block 561 len 20480 12/30 Read block 68 len 28672 13/30 Read block 151 len 8192 14/30 Read block 180 len 36864 15/30 Read block 509 len 4096 16/30 Write block 404 len 8192 data: 12 9 17/30 Write block 39 len 28672 data: 10 10 12 14 15 13 14 18/30 Read block 245 len 4096 19/30 Read block 524 len 4096 20/30 Read block 177 len 36864 21/30 Read block 194 len 36864 22/30 Write block 276 len 24576 data: 7 7 6 8 7 7 23/30 Read block 185 len 20480 24/30 Read block 270 len 4096 25/30 Write block 156 len 24576 data: 6 6 5 5 3 5 26/30 Read block 118 len 40960 27/30 Read block 588 len 8192 28/30 Write block 456 len 20480 data: 11 9 7 8 7 29/30 Read block 8 len 20480 30/30 Write block 65 len 20480 data: 10 11 11 11 8 ---------------------------------------------------------------- Crucible gen:116 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Read 7 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Write 7 Done Done Done false 1004 Acked Write 9 Done Sent Done false 1005 Acked Write 9 Done Sent Done false 1006 Acked Write 10 Done Sent Done false 1007 Acked Read 10 Done Sent Done false 1008 Acked Read 3 Done Sent Done false 1009 Acked Read 3 Done Sent Done false 1010 Acked Read 5 Done Sent Done false 1011 Acked Read 7 Done Sent Done false 1012 Acked Read 2 Done Sent Done false 1013 Acked Read 9 Done Sent Done false 1014 Acked Read 1 Done Sent Done false 1015 Acked Write 2 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Read 1 Done Sent Done false 1018 Acked Read 1 Done Sent Done false 1019 Acked Read 9 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Write 6 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 1 Done Sent Done false 1024 Acked Write 6 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Read 2 Done Sent Done false 1027 Acked Write 5 Done Sent Done false 1028 Acked Read 5 Done Sent Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 26 1 28 Done 29 4 29 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:25.655845817Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:25.656050009Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1597,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 192 192 192 F F F 1 020-039 112 112 112 194 194 194 T F T <--- 2 040-059 113 113 113 195 195 195 T F T <--- 3 060-079 114 114 114 198 198 198 T F T <--- 4 080-099 114 114 114 198 198 198 T F T <--- 5 100-119 115 115 115 199 199 199 F F F 6 120-139 115 115 115 199 199 199 F F F 7 140-159 110 110 110 189 189 189 T F T <--- 8 160-179 115 115 115 199 199 199 T T T 9 180-199 114 114 114 197 197 197 F F F 10 200-219 113 113 113 195 195 195 F F F 11 220-239 112 112 112 193 193 193 F F F 12 240-259 114 114 114 196 196 196 F F F 13 260-279 114 114 114 196 196 196 T F T <--- 14 280-299 113 113 113 195 195 195 T F T <--- 15 300-319 115 115 115 199 199 199 F F F 16 320-339 111 111 111 192 192 192 F F F 17 340-359 114 114 114 198 198 198 F F F 18 360-379 114 114 114 196 196 196 T F T <--- 19 380-399 111 111 111 192 192 192 F F F 20 400-419 111 111 111 192 192 192 T F T <--- 21 420-439 115 115 115 199 199 199 F F F 22 440-459 115 115 115 199 199 199 T F T <--- 23 460-479 100 100 100 175 175 175 T F T <--- 24 480-499 111 111 111 192 192 192 T T T 25 500-519 111 111 111 192 192 192 F F F 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 113 113 113 195 195 195 F F F 29 580-599 115 115 115 199 199 199 F F F Max gen: 115, Max flush: 199 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 58 Verifying data now No change after verify Loop: 58 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 192 192 192 F F F 1 020-039 116 116 116 200 200 200 F F F 2 040-059 116 116 116 200 200 200 F F F 3 060-079 116 116 116 200 200 200 F F F 4 080-099 116 116 116 200 200 200 F F F 5 100-119 115 115 115 199 199 199 F F F 6 120-139 115 115 115 199 199 199 F F F 7 140-159 116 116 116 200 200 200 F F F 8 160-179 116 116 116 200 200 200 F F F 9 180-199 114 114 114 197 197 197 F F F 10 200-219 113 113 113 195 195 195 F F F 11 220-239 112 112 112 193 193 193 F F F 12 240-259 114 114 114 196 196 196 F F F 13 260-279 116 116 116 200 200 200 F F F 14 280-299 116 116 116 200 200 200 F F F 15 300-319 115 115 115 199 199 199 F F F 16 320-339 111 111 111 192 192 192 F F F 17 340-359 114 114 114 198 198 198 F F F 18 360-379 116 116 116 200 200 200 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 116 116 116 200 200 200 F F F 21 420-439 115 115 115 199 199 199 F F F 22 440-459 116 116 116 200 200 200 F F F 23 460-479 116 116 116 200 200 200 F F F 24 480-499 116 116 116 200 200 200 F F F 25 500-519 111 111 111 192 192 192 F F F 26 520-539 102 102 102 179 179 179 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 113 113 113 195 195 195 F F F 29 580-599 115 115 115 199 199 199 F F F Max gen: 116, Max flush: 200 Begin loop 59 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.911328652Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:29.918484752Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.918534322Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.925465624Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.925738813Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.925767648Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.925776016Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.925787554Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605} {"msg":"Crucible b6eb2d06-684c-4592-a819-bb1e04eb3f1e has session id: fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926151466Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"Upstairs opts: Upstairs UUID: b6eb2d06-684c-4592-a819-bb1e04eb3f1e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.92618101Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"Crucible stats registered with UUID: b6eb2d06-684c-4592-a819-bb1e04eb3f1e","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926194688Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:29.926204116Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926354178Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"io task","client":"1","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926383632Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"io task","client":"0","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.92639453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"io task","client":"2","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926505899Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"task":"crutest"} {"msg":"The guest has requested activation with gen:118","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926521936Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605} {"msg":"b6eb2d06-684c-4592-a819-bb1e04eb3f1e active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.92655719Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926566718Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"client":"0","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926576556Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"client":"1","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926586204Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"client":"2","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926595623Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"io task","client":"2","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926605491Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"io task","client":"0","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926615409Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"io task","client":"1","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"ds_connection connected from Ok(127.0.0.1:50830)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926689665Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"io task","client":"2","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"ds_connection connected from Ok(127.0.0.1:61667)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926703163Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"io task","client":"1","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"ds_connection connected from Ok(127.0.0.1:55881)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.926722219Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"io task","client":"0","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927378536Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"client":"1","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927404261Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"client":"2","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927496444Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"client":"0","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927651675Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927668872Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"[0]R flush_numbers[0..12]: [192, 200, 200, 200, 200, 199, 199, 200, 200, 197, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927778351Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"downstairs","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"[0]R generation[0..12]: [111, 116, 116, 116, 116, 115, 115, 116, 116, 114, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927789659Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"downstairs","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927800497Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"downstairs","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"[1]R flush_numbers[0..12]: [192, 200, 200, 200, 200, 199, 199, 200, 200, 197, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927811075Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"downstairs","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"[1]R generation[0..12]: [111, 116, 116, 116, 116, 115, 115, 116, 116, 114, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927821523Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"downstairs","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927832431Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"downstairs","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"[2]R flush_numbers[0..12]: [192, 200, 200, 200, 200, 199, 199, 200, 200, 197, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927842959Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"downstairs","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"[2]R generation[0..12]: [111, 116, 116, 116, 116, 115, 115, 116, 116, 114, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927853367Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"downstairs","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927866395Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"downstairs","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"Max found gen is 117","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927875943Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"downstairs","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"Generation requested: 118 >= found:117","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927885571Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"downstairs","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"Next flush: 201","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927894829Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"downstairs","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927904138Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"":"downstairs","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927913846Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927922604Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"client":"0","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927932162Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"client":"1","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927941681Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"client":"2","":"downstairs","session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927951449Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927960467Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"b6eb2d06-684c-4592-a819-bb1e04eb3f1e is now active with session: fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927970395Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927979034Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"session_id":"fbdd61d1-1b2c-4e9f-bd2f-ac6309a00b2b"} {"msg":"The guest has finished waiting for activation with:118","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.927988132Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605} {"msg":"Activated sub_volume b6eb2d06-684c-4592-a819-bb1e04eb3f1e","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.92799613Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.928004249Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.928281017Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"task":"crutest"} 01/30 Write block 590 len 28672 data: 14 12 14 16 15 13 10 02/30 Flush 03/30 Write block 101 len 20480 data: 7 7 9 8 9 04/30 Write block 84 len 12288 data: 8 10 9 05/30 Read block 343 len 20480 06/30 Flush 07/30 Write block 411 len 24576 data: 6 6 5 8 8 10 08/30 Read block 337 len 16384 09/30 Read block 101 len 28672 10/30 Write block 184 len 40960 data: 9 8 9 6 6 8 9 12 14 17 11/30 Read block 272 len 12288 12/30 Write block 532 len 12288 data: 6 6 7 13/30 Read block 341 len 28672 14/30 Read block 433 len 4096 15/30 Write block 193 len 16384 data: 18 15 15 15 16/30 Read block 314 len 12288 17/30 Write block 185 len 24576 data: 9 10 7 7 9 10 18/30 Read block 68 len 28672 19/30 Write block 327 len 4096 data: 9 20/30 Write block 29 len 40960 data: 12 15 15 15 12 10 9 7 8 10 21/30 Write block 255 len 24576 data: 10 9 10 13 12 14 22/30 Read block 76 len 36864 23/30 Write block 411 len 4096 data: 7 24/30 Flush 25/30 Write block 69 len 16384 data: 9 6 6 7 26/30 Write block 300 len 12288 data: 10 10 13 27/30 Write block 38 len 40960 data: 11 11 11 13 15 16 14 15 11 9 28/30 Read block 110 len 32768 29/30 Read block 367 len 32768 30/30 Write block 343 len 20480 data: 10 10 9 8 5 ---------------------------------------------------------------- Crucible gen:118 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 5 Sent Done Done false 1003 Acked Write 3 Sent Done Done false 1004 Acked Read 5 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 6 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Read 7 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Read 3 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Read 7 Sent Done Done false 1013 Acked Read 1 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Write 6 Sent Done Done false 1017 Acked Read 7 Sent Done Done false 1018 Acked Write 1 Sent Done Done false 1019 Acked Write 10 Sent Done Done false 1020 Acked Write 6 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 4 Sent Done Done false 1025 Acked Write 3 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Read 8 Sent Done Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 1 1 30 Done 0 27 27 54 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1001)) Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:29.937514598Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:29.937688425Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1605,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 192 192 192 F F F 1 020-039 116 118 118 200 203 203 F T T <--- 2 040-059 116 116 116 200 200 200 F T T <--- 3 060-079 116 116 116 200 200 200 F T T <--- 4 080-099 116 118 118 200 202 202 F F F <--- 5 100-119 115 118 118 199 202 202 F F F <--- 6 120-139 115 115 115 199 199 199 F F F 7 140-159 116 116 116 200 200 200 F F F 8 160-179 116 116 116 200 200 200 F F F 9 180-199 114 118 118 197 203 203 F F F <--- 10 200-219 113 113 113 195 195 195 F F F 11 220-239 112 112 112 193 193 193 F F F 12 240-259 114 118 118 196 203 203 F F F <--- 13 260-279 116 118 118 200 203 203 F F F <--- 14 280-299 116 116 116 200 200 200 F F F 15 300-319 115 115 115 199 199 199 F T T <--- 16 320-339 111 118 118 192 203 203 F F F <--- 17 340-359 114 114 114 198 198 198 F T T <--- 18 360-379 116 116 116 200 200 200 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 116 118 118 200 203 203 F F F <--- 21 420-439 115 115 115 199 199 199 F F F 22 440-459 116 116 116 200 200 200 F F F 23 460-479 116 116 116 200 200 200 F F F 24 480-499 116 116 116 200 200 200 F F F 25 500-519 111 111 111 192 192 192 F F F 26 520-539 102 118 118 179 203 203 F F F <--- 27 540-559 100 100 100 175 175 175 F F F 28 560-579 113 113 113 195 195 195 F F F 29 580-599 118 118 118 201 201 201 F F F Max gen: 118, Max flush: 203 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 59 Verifying data now No change after verify Loop: 59 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 192 192 192 F F F 1 020-039 119 119 119 204 204 204 F F F 2 040-059 119 119 119 204 204 204 F F F 3 060-079 119 119 119 204 204 204 F F F 4 080-099 118 118 118 202 202 202 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 115 115 115 199 199 199 F F F 7 140-159 116 116 116 200 200 200 F F F 8 160-179 116 116 116 200 200 200 F F F 9 180-199 118 118 118 203 203 203 F F F 10 200-219 113 113 113 195 195 195 F F F 11 220-239 112 112 112 193 193 193 F F F 12 240-259 118 118 118 203 203 203 F F F 13 260-279 118 118 118 203 203 203 F F F 14 280-299 116 116 116 200 200 200 F F F 15 300-319 119 119 119 204 204 204 F F F 16 320-339 118 118 118 203 203 203 F F F 17 340-359 119 119 119 204 204 204 F F F 18 360-379 116 116 116 200 200 200 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 118 118 118 203 203 203 F F F 21 420-439 115 115 115 199 199 199 F F F 22 440-459 116 116 116 200 200 200 F F F 23 460-479 116 116 116 200 200 200 F F F 24 480-499 116 116 116 200 200 200 F F F 25 500-519 111 111 111 192 192 192 F F F 26 520-539 118 118 118 203 203 203 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 113 113 113 195 195 195 F F F 29 580-599 118 118 118 201 201 201 F F F Max gen: 119, Max flush: 204 Begin loop 60 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.196718815Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.204962571Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.2052928Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.205320134Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.205329103Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.205340841Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613} {"msg":"Crucible 019036ce-9f73-4ce1-a164-cc0718367750 has session id: 31e53891-bf76-4453-9399-9df7b0034e31","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.205676148Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"Upstairs opts: Upstairs UUID: 019036ce-9f73-4ce1-a164-cc0718367750, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.205705552Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"Crucible stats registered with UUID: 019036ce-9f73-4ce1-a164-cc0718367750","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.2057167Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:34.205726038Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.20587515Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"io task","client":"1","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.205891707Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"io task","client":"0","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.205901825Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"io task","client":"2","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.20603777Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"task":"crutest"} {"msg":"The guest has requested activation with gen:120","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.206054517Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613} {"msg":"019036ce-9f73-4ce1-a164-cc0718367750 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.206087351Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.206096689Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"client":"0","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.206106357Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"client":"1","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.206115885Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"client":"2","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.206125434Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"io task","client":"2","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.206135252Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"io task","client":"0","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.20614501Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"io task","client":"1","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"ds_connection connected from Ok(127.0.0.1:59467)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.206192221Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"io task","client":"1","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"ds_connection connected from Ok(127.0.0.1:62874)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.206203109Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"io task","client":"0","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"ds_connection connected from Ok(127.0.0.1:33114)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.206244131Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"io task","client":"2","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.206850348Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"client":"0","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.206875293Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"client":"2","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.206888941Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"client":"1","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207152651Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207162789Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"[0]R flush_numbers[0..12]: [192, 204, 204, 204, 202, 202, 199, 200, 200, 203, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207194383Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"downstairs","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"[0]R generation[0..12]: [111, 119, 119, 119, 118, 118, 115, 116, 116, 118, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207205631Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"downstairs","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207216589Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"downstairs","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"[1]R flush_numbers[0..12]: [192, 204, 204, 204, 202, 202, 199, 200, 200, 203, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207227037Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"downstairs","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"[1]R generation[0..12]: [111, 119, 119, 119, 118, 118, 115, 116, 116, 118, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207237375Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"downstairs","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207248203Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"downstairs","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"[2]R flush_numbers[0..12]: [192, 204, 204, 204, 202, 202, 199, 200, 200, 203, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207258541Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"downstairs","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"[2]R generation[0..12]: [111, 119, 119, 119, 118, 118, 115, 116, 116, 118, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207269039Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"downstairs","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207280027Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"downstairs","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"Max found gen is 120","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207289356Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"downstairs","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"Generation requested: 120 >= found:120","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207299014Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"downstairs","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"Next flush: 205","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207308272Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"downstairs","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.20731767Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"":"downstairs","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207328978Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207337767Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"client":"0","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207347295Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"client":"1","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207356783Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"client":"2","":"downstairs","session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207366461Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207375469Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"019036ce-9f73-4ce1-a164-cc0718367750 is now active with session: 31e53891-bf76-4453-9399-9df7b0034e31","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207385218Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207393986Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"session_id":"31e53891-bf76-4453-9399-9df7b0034e31"} {"msg":"The guest has finished waiting for activation with:120","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207402974Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613} {"msg":"Activated sub_volume 019036ce-9f73-4ce1-a164-cc0718367750","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207410943Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207419001Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.207785123Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"task":"crutest"} 01/30 Write block 325 len 8192 data: 10 10 02/30 Read block 522 len 36864 03/30 Read block 352 len 40960 04/30 Read block 367 len 8192 05/30 Flush 06/30 Write block 351 len 12288 data: 10 9 9 07/30 Write block 334 len 16384 data: 11 11 11 10 08/30 Write block 515 len 28672 data: 11 11 11 9 9 9 11 09/30 Write block 582 len 32768 data: 13 9 7 6 8 11 12 13 10/30 Read block 219 len 16384 11/30 Read block 411 len 4096 12/30 Read block 166 len 36864 13/30 Read block 215 len 20480 14/30 Write block 580 len 36864 data: 12 13 14 10 8 7 9 12 13 15/30 Write block 25 len 4096 data: 9 16/30 Flush 17/30 Write block 499 len 40960 data: 9 10 8 9 10 11 10 8 9 5 18/30 Read block 22 len 20480 19/30 Read block 127 len 16384 20/30 Flush 21/30 Write block 457 len 28672 data: 10 8 9 8 4 4 4 22/30 Flush 23/30 Write block 11 len 4096 data: 18 24/30 Read block 252 len 4096 25/30 Write block 155 len 20480 data: 6 7 7 6 6 26/30 Write block 344 len 12288 data: 11 10 9 27/30 Read block 92 len 4096 28/30 Write block 442 len 32768 data: 9 9 8 8 6 6 4 6 29/30 Read block 141 len 16384 30/30 Read block 2 len 4096 ---------------------------------------------------------------- Crucible gen:120 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Write 3 Done Done Done false 1006 Acked Write 4 Done Sent Done false 1007 Acked Write 7 Done Sent Done false 1008 Acked Write 8 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Write 9 Done Sent Done false 1014 Acked Write 1 Done Sent Done false 1015 Acked Flush 0 Done Sent Done false 1016 Acked Write 10 Done Sent Done false 1017 Acked Read 5 Done Sent Done false 1018 Acked Read 4 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 7 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 1 Done Sent Done false 1023 Acked Read 1 Done Sent Done false 1024 Acked Write 5 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Read 1 Done Sent Done false 1027 Acked Write 8 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Read 1 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 24 0 24 Done 25 1 25 51 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1004)) Some(JobId(1021)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:34.216247868Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:34.216432143Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1613,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 111 111 111 192 192 192 T F T <--- 1 020-039 120 119 120 206 204 206 F F F <--- 2 040-059 119 119 119 204 204 204 F F F 3 060-079 119 119 119 204 204 204 F F F 4 080-099 118 118 118 202 202 202 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 115 115 115 199 199 199 F F F 7 140-159 116 116 116 200 200 200 T F T <--- 8 160-179 116 116 116 200 200 200 F F F 9 180-199 118 118 118 203 203 203 F F F 10 200-219 113 113 113 195 195 195 F F F 11 220-239 112 112 112 193 193 193 F F F 12 240-259 118 118 118 203 203 203 F F F 13 260-279 118 118 118 203 203 203 F F F 14 280-299 116 116 116 200 200 200 F F F 15 300-319 119 119 119 204 204 204 F F F 16 320-339 120 120 120 206 205 206 F F F <--- 17 340-359 120 119 120 206 204 206 T T T <--- 18 360-379 116 116 116 200 200 200 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 118 118 118 203 203 203 F F F 21 420-439 115 115 115 199 199 199 F F F 22 440-459 120 116 120 208 200 208 T F T <--- 23 460-479 120 116 120 208 200 208 F F F <--- 24 480-499 120 116 120 207 200 207 F F F <--- 25 500-519 120 111 120 207 192 207 F F F <--- 26 520-539 120 118 120 206 203 206 F F F <--- 27 540-559 100 100 100 175 175 175 F F F 28 560-579 113 113 113 195 195 195 F F F 29 580-599 120 118 120 206 201 206 F F F <--- Max gen: 120, Max flush: 208 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 60 Verifying data now No change after verify Loop: 60 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 209 209 209 F F F 1 020-039 120 120 120 206 206 206 F F F 2 040-059 119 119 119 204 204 204 F F F 3 060-079 119 119 119 204 204 204 F F F 4 080-099 118 118 118 202 202 202 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 115 115 115 199 199 199 F F F 7 140-159 121 121 121 209 209 209 F F F 8 160-179 116 116 116 200 200 200 F F F 9 180-199 118 118 118 203 203 203 F F F 10 200-219 113 113 113 195 195 195 F F F 11 220-239 112 112 112 193 193 193 F F F 12 240-259 118 118 118 203 203 203 F F F 13 260-279 118 118 118 203 203 203 F F F 14 280-299 116 116 116 200 200 200 F F F 15 300-319 119 119 119 204 204 204 F F F 16 320-339 120 120 120 206 206 206 F F F 17 340-359 121 121 121 209 209 209 F F F 18 360-379 116 116 116 200 200 200 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 118 118 118 203 203 203 F F F 21 420-439 115 115 115 199 199 199 F F F 22 440-459 121 121 121 209 209 209 F F F 23 460-479 120 120 120 208 208 208 F F F 24 480-499 120 120 120 207 207 207 F F F 25 500-519 120 120 120 207 207 207 F F F 26 520-539 120 120 120 206 206 206 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 113 113 113 195 195 195 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 121, Max flush: 209 Begin loop 61 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.464949593Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:38.47212078Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.472163422Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.479019008Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.47927586Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.479305544Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.479316822Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.47932837Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621} {"msg":"Crucible 18c3f1fd-563e-4f74-bafd-0dad897c5c05 has session id: fd295795-f060-4ac5-a82c-6a590b713e75","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.479669516Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"Upstairs opts: Upstairs UUID: 18c3f1fd-563e-4f74-bafd-0dad897c5c05, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.47969843Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"Crucible stats registered with UUID: 18c3f1fd-563e-4f74-bafd-0dad897c5c05","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.479709778Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:38.479719307Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.479866049Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"io task","client":"1","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.479882996Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"io task","client":"0","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.479892964Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"io task","client":"2","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.480017011Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"task":"crutest"} {"msg":"The guest has requested activation with gen:122","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.480030278Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621} {"msg":"18c3f1fd-563e-4f74-bafd-0dad897c5c05 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.480053754Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.480063222Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"client":"0","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.48007318Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"client":"1","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.480084558Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"client":"2","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.480094166Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"io task","client":"2","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.480104224Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"io task","client":"0","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.480114213Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"io task","client":"1","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"ds_connection connected from Ok(127.0.0.1:54668)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.480150186Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"io task","client":"0","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"ds_connection connected from Ok(127.0.0.1:59585)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.480160484Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"io task","client":"1","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"ds_connection connected from Ok(127.0.0.1:59314)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.480227751Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"io task","client":"2","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.48087775Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"client":"2","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.480900885Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"client":"1","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.480912503Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"client":"0","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481217946Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481227974Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"[0]R flush_numbers[0..12]: [209, 206, 204, 204, 202, 202, 199, 209, 200, 203, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481263597Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"downstairs","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"[0]R generation[0..12]: [121, 120, 119, 119, 118, 118, 115, 121, 116, 118, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481274985Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"downstairs","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481288173Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"downstairs","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"[1]R flush_numbers[0..12]: [209, 206, 204, 204, 202, 202, 199, 209, 200, 203, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481298751Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"downstairs","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"[1]R generation[0..12]: [121, 120, 119, 119, 118, 118, 115, 121, 116, 118, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481309069Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"downstairs","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481319777Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"downstairs","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"[2]R flush_numbers[0..12]: [209, 206, 204, 204, 202, 202, 199, 209, 200, 203, 195, 193]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481330215Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"downstairs","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"[2]R generation[0..12]: [121, 120, 119, 119, 118, 118, 115, 121, 116, 118, 113, 112]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481340573Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"downstairs","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481351321Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"downstairs","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"Max found gen is 122","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481360509Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"downstairs","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"Generation requested: 122 >= found:122","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481370187Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"downstairs","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"Next flush: 210","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481379386Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"downstairs","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481388764Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"":"downstairs","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481398392Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.4814069Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"client":"0","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481416789Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"client":"1","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481426407Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"client":"2","":"downstairs","session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481437215Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481446163Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"18c3f1fd-563e-4f74-bafd-0dad897c5c05 is now active with session: fd295795-f060-4ac5-a82c-6a590b713e75","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481456031Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.48146456Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"session_id":"fd295795-f060-4ac5-a82c-6a590b713e75"} {"msg":"The guest has finished waiting for activation with:122","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481473528Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621} {"msg":"Activated sub_volume 18c3f1fd-563e-4f74-bafd-0dad897c5c05","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481481476Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481489605Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.481792918Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"task":"crutest"} 01/30 Write block 300 len 20480 data: 11 11 14 15 17 02/30 Read block 414 len 24576 03/30 Flush 04/30 Write block 24 len 20480 data: 9 10 9 7 10 05/30 Write block 441 len 12288 data: 8 10 10 06/30 Read block 43 len 32768 07/30 Read block 545 len 16384 08/30 Write block 260 len 16384 data: 15 15 13 12 09/30 Read block 593 len 16384 10/30 Read block 397 len 36864 11/30 Write block 260 len 8192 data: 16 16 12/30 Read block 399 len 28672 13/30 Read block 218 len 8192 14/30 Read block 232 len 8192 15/30 Read block 482 len 8192 16/30 Flush 17/30 Write block 233 len 4096 data: 11 18/30 Read block 501 len 36864 19/30 Read block 562 len 28672 20/30 Read block 559 len 8192 21/30 Read block 196 len 36864 22/30 Write block 310 len 40960 data: 13 13 9 9 10 10 11 12 9 9 23/30 Read block 57 len 24576 24/30 Read block 18 len 24576 25/30 Read block 300 len 32768 26/30 Write block 214 len 8192 data: 16 16 27/30 Read block 211 len 12288 28/30 Write block 77 len 40960 data: 6 7 6 5 6 8 5 9 11 10 29/30 Write block 255 len 32768 data: 11 10 11 14 13 17 17 14 30/30 Read block 79 len 20480 ---------------------------------------------------------------- Crucible gen:122 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 5 Done Done Done false 1004 Acked Write 3 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Read 4 Sent Done Done false 1007 Acked Write 4 Sent Done Done false 1008 Acked Read 4 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Write 2 Sent Done Done false 1011 Acked Read 7 Sent Done Done false 1012 Acked Read 2 Sent Done Done false 1013 Acked Read 2 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Read 7 Sent Done Done false 1019 Acked Read 2 Sent Done Done false 1020 Acked Read 9 Sent Done Done false 1021 Acked Write 10 Sent Done Done false 1022 Acked Read 6 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Read 8 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Read 3 Sent Done Done false 1027 Acked Write 10 Sent Done Done false 1028 Acked Write 8 Sent Done Done false 1029 Acked Read 5 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 26 0 0 26 Done 1 27 27 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1002)) Some(JobId(1015)) Some(JobId(1015)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:38.490401746Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:38.490599079Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1621,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 209 209 209 F F F 1 020-039 120 122 122 206 211 211 T F F <--- 2 040-059 119 119 119 204 204 204 F F F 3 060-079 119 119 119 204 204 204 F T T <--- 4 080-099 118 118 118 202 202 202 F T T <--- 5 100-119 118 118 118 202 202 202 F F F 6 120-139 115 115 115 199 199 199 F F F 7 140-159 121 121 121 209 209 209 F F F 8 160-179 116 116 116 200 200 200 F F F 9 180-199 118 118 118 203 203 203 F F F 10 200-219 113 113 113 195 195 195 F T T <--- 11 220-239 112 112 112 193 193 193 F T T <--- 12 240-259 118 118 118 203 203 203 F T T <--- 13 260-279 118 122 122 203 211 211 F T T <--- 14 280-299 116 116 116 200 200 200 F F F 15 300-319 122 122 122 210 210 210 F T T <--- 16 320-339 120 120 120 206 206 206 F F F 17 340-359 121 121 121 209 209 209 F F F 18 360-379 116 116 116 200 200 200 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 118 118 118 203 203 203 F F F 21 420-439 115 115 115 199 199 199 F F F 22 440-459 121 122 122 209 211 211 F F F <--- 23 460-479 120 120 120 208 208 208 F F F 24 480-499 120 120 120 207 207 207 F F F 25 500-519 120 120 120 207 207 207 F F F 26 520-539 120 120 120 206 206 206 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 113 113 113 195 195 195 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 122, Max flush: 211 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 61 Verifying data now No change after verify Loop: 61 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 209 209 209 F F F 1 020-039 122 122 122 211 211 211 F F F 2 040-059 119 119 119 204 204 204 F F F 3 060-079 123 123 123 212 212 212 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 115 115 115 199 199 199 F F F 7 140-159 121 121 121 209 209 209 F F F 8 160-179 116 116 116 200 200 200 F F F 9 180-199 118 118 118 203 203 203 F F F 10 200-219 123 123 123 212 212 212 F F F 11 220-239 123 123 123 212 212 212 F F F 12 240-259 123 123 123 212 212 212 F F F 13 260-279 123 123 123 212 212 212 F F F 14 280-299 116 116 116 200 200 200 F F F 15 300-319 123 123 123 212 212 212 F F F 16 320-339 120 120 120 206 206 206 F F F 17 340-359 121 121 121 209 209 209 F F F 18 360-379 116 116 116 200 200 200 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 118 118 118 203 203 203 F F F 21 420-439 115 115 115 199 199 199 F F F 22 440-459 122 122 122 211 211 211 F F F 23 460-479 120 120 120 208 208 208 F F F 24 480-499 120 120 120 207 207 207 F F F 25 500-519 120 120 120 207 207 207 F F F 26 520-539 120 120 120 206 206 206 F F F 27 540-559 100 100 100 175 175 175 F F F 28 560-579 113 113 113 195 195 195 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 123, Max flush: 212 Begin loop 62 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.741072772Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:42.748295849Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.748347429Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.755129379Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.755406617Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.755435132Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.75544374Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.755455308Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629} {"msg":"Crucible 8a125ef0-05aa-4118-a752-dec95f8a267a has session id: 88e40e4f-15c4-4ec0-a358-a413d1e9789c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.755788936Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"Upstairs opts: Upstairs UUID: 8a125ef0-05aa-4118-a752-dec95f8a267a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.755810752Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"Crucible stats registered with UUID: 8a125ef0-05aa-4118-a752-dec95f8a267a","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.75582058Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:42.755857173Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756002816Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"io task","client":"0","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756019213Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"io task","client":"1","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756029071Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"io task","client":"2","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756196879Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"task":"crutest"} {"msg":"The guest has requested activation with gen:124","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756214036Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629} {"msg":"8a125ef0-05aa-4118-a752-dec95f8a267a active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756249769Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756259348Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"client":"0","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756268956Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"client":"1","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756278464Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"client":"2","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756288102Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"io task","client":"2","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.75629796Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"io task","client":"0","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756308059Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"io task","client":"1","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"ds_connection connected from Ok(127.0.0.1:57214)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756402841Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"io task","client":"2","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"ds_connection connected from Ok(127.0.0.1:49968)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756555542Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"io task","client":"1","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"ds_connection connected from Ok(127.0.0.1:35901)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.756581337Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"io task","client":"0","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757069936Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"client":"1","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757087573Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"client":"0","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.75709844Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"client":"2","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757424149Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757444096Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"[0]R flush_numbers[0..12]: [209, 211, 204, 212, 212, 202, 199, 209, 200, 203, 212, 212]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.75747416Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"downstairs","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"[0]R generation[0..12]: [121, 122, 119, 123, 123, 118, 115, 121, 116, 118, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757485958Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"downstairs","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757497086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"downstairs","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"[1]R flush_numbers[0..12]: [209, 211, 204, 212, 212, 202, 199, 209, 200, 203, 212, 212]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757507764Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"downstairs","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"[1]R generation[0..12]: [121, 122, 119, 123, 123, 118, 115, 121, 116, 118, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757518492Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"downstairs","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.75752941Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"downstairs","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"[2]R flush_numbers[0..12]: [209, 211, 204, 212, 212, 202, 199, 209, 200, 203, 212, 212]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757542607Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"downstairs","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"[2]R generation[0..12]: [121, 122, 119, 123, 123, 118, 115, 121, 116, 118, 123, 123]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757553215Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"downstairs","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757564183Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"downstairs","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"Max found gen is 124","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757573611Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"downstairs","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"Generation requested: 124 >= found:124","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.75758337Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"downstairs","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"Next flush: 213","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757592678Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"downstairs","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757602016Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"":"downstairs","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757611764Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757620543Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"client":"0","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757630471Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"client":"1","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757639969Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"client":"2","":"downstairs","session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757649817Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757658866Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"8a125ef0-05aa-4118-a752-dec95f8a267a is now active with session: 88e40e4f-15c4-4ec0-a358-a413d1e9789c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757668884Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757677672Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"session_id":"88e40e4f-15c4-4ec0-a358-a413d1e9789c"} {"msg":"The guest has finished waiting for activation with:124","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.75768823Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629} {"msg":"Activated sub_volume 8a125ef0-05aa-4118-a752-dec95f8a267a","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757696448Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.757704717Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.758023707Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"task":"crutest"} 01/30 Write block 340 len 28672 data: 9 8 10 11 12 11 10 02/30 Write block 574 len 20480 data: 5 8 8 10 12 03/30 Read block 46 len 40960 04/30 Flush 05/30 Flush 06/30 Flush 07/30 Write block 242 len 32768 data: 10 10 9 11 11 12 11 12 08/30 Write block 141 len 24576 data: 16 13 10 10 9 7 09/30 Read block 570 len 24576 10/30 Read block 218 len 8192 11/30 Write block 285 len 4096 data: 6 12/30 Write block 468 len 12288 data: 6 6 8 13/30 Write block 545 len 24576 data: 11 10 9 9 8 8 14/30 Read block 264 len 16384 15/30 Read block 500 len 8192 16/30 Read block 482 len 16384 17/30 Read block 55 len 12288 18/30 Write block 203 len 32768 data: 8 7 7 7 8 8 11 13 19/30 Write block 545 len 20480 data: 12 11 10 10 9 20/30 Write block 312 len 32768 data: 10 10 11 11 12 13 10 10 21/30 Write block 333 len 32768 data: 11 12 12 12 11 12 10 10 22/30 Read block 363 len 32768 23/30 Write block 26 len 32768 data: 10 8 11 13 16 16 16 13 24/30 Flush 25/30 Write block 464 len 8192 data: 6 4 26/30 Read block 349 len 8192 27/30 Write block 275 len 12288 data: 7 8 8 28/30 Write block 334 len 20480 data: 13 13 13 12 13 29/30 Write block 325 len 16384 data: 11 11 10 7 30/30 Read block 37 len 40960 ---------------------------------------------------------------- Crucible gen:124 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 5 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Flush 0 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Write 8 Sent Done Done false 1007 Acked Write 6 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Read 2 Sent Done Done false 1010 Acked Write 1 Sent Done Done false 1011 Acked Write 3 Sent Done Done false 1012 Acked Write 6 Sent Done Done false 1013 Acked Read 4 Sent Done Done false 1014 Acked Read 2 Sent Done Done false 1015 Acked Read 4 Sent Done Done false 1016 Acked Read 3 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Write 5 Sent Done Done false 1019 Acked Write 8 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Read 8 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Flush 0 Sent Done Done false 1024 Acked Write 2 Sent Done Done false 1025 Acked Read 2 Sent Done Done false 1026 Acked Write 3 Sent Done Done false 1027 Acked Write 5 Sent Done Done false 1028 Acked Write 4 Sent Done Done false 1029 Acked Read 10 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 0 0 27 Done 3 30 30 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1023)) Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:42.766398339Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:42.766639804Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1629,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 209 209 209 F F F 1 020-039 122 124 124 211 216 216 F F F <--- 2 040-059 119 119 119 204 204 204 F F F 3 060-079 123 123 123 212 212 212 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 115 115 115 199 199 199 F F F 7 140-159 121 124 124 209 216 216 F F F <--- 8 160-179 116 116 116 200 200 200 F F F 9 180-199 118 118 118 203 203 203 F F F 10 200-219 123 124 124 212 216 216 F F F <--- 11 220-239 123 123 123 212 212 212 F F F 12 240-259 123 124 124 212 216 216 F F F <--- 13 260-279 123 123 123 212 212 212 F T T <--- 14 280-299 116 124 124 200 216 216 F F F <--- 15 300-319 123 124 124 212 216 216 F F F <--- 16 320-339 120 124 124 206 216 216 F T T <--- 17 340-359 121 124 124 209 216 216 T F F <--- 18 360-379 116 116 116 200 200 200 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 118 118 118 203 203 203 F F F 21 420-439 115 115 115 199 199 199 F F F 22 440-459 122 122 122 211 211 211 F F F 23 460-479 120 124 124 208 216 216 F T T <--- 24 480-499 120 120 120 207 207 207 F F F 25 500-519 120 120 120 207 207 207 F F F 26 520-539 120 120 120 206 206 206 F F F 27 540-559 100 124 124 175 216 216 F F F <--- 28 560-579 113 124 124 195 213 213 T F F <--- 29 580-599 120 120 120 206 206 206 F F F Max gen: 124, Max flush: 216 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 62 Verifying data now No change after verify Loop: 62 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 209 209 209 F F F 1 020-039 124 124 124 216 216 216 F F F 2 040-059 119 119 119 204 204 204 F F F 3 060-079 123 123 123 212 212 212 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 115 115 115 199 199 199 F F F 7 140-159 124 124 124 216 216 216 F F F 8 160-179 116 116 116 200 200 200 F F F 9 180-199 118 118 118 203 203 203 F F F 10 200-219 124 124 124 216 216 216 F F F 11 220-239 123 123 123 212 212 212 F F F 12 240-259 124 124 124 216 216 216 F F F 13 260-279 125 125 125 217 217 217 F F F 14 280-299 124 124 124 216 216 216 F F F 15 300-319 124 124 124 216 216 216 F F F 16 320-339 125 125 125 217 217 217 F F F 17 340-359 124 124 124 216 216 216 F F F 18 360-379 116 116 116 200 200 200 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 118 118 118 203 203 203 F F F 21 420-439 115 115 115 199 199 199 F F F 22 440-459 122 122 122 211 211 211 F F F 23 460-479 125 125 125 217 217 217 F F F 24 480-499 120 120 120 207 207 207 F F F 25 500-519 120 120 120 207 207 207 F F F 26 520-539 120 120 120 206 206 206 F F F 27 540-559 124 124 124 216 216 216 F F F 28 560-579 124 124 124 213 213 213 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 125, Max flush: 217 Begin loop 63 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.023735576Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:47.030897874Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.030954094Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.037710029Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.037981538Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038010522Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038019051Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038030559Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637} {"msg":"Crucible e2ca5540-7b86-4a33-9ade-6164334af80c has session id: 02e9353e-0c4a-405e-9dbb-51a47ef604bd","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038429704Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"Upstairs opts: Upstairs UUID: e2ca5540-7b86-4a33-9ade-6164334af80c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038459178Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"Crucible stats registered with UUID: e2ca5540-7b86-4a33-9ade-6164334af80c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038472066Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:47.038481294Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038605271Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"io task","client":"1","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038622338Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"io task","client":"0","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038632266Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"io task","client":"2","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038791266Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"task":"crutest"} {"msg":"The guest has requested activation with gen:126","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038815582Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637} {"msg":"e2ca5540-7b86-4a33-9ade-6164334af80c active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038832678Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038841787Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"client":"0","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038851535Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"client":"1","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038861223Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"client":"2","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038870791Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"io task","client":"2","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038880849Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"io task","client":"0","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038890818Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"io task","client":"1","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:44343)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038947737Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"io task","client":"1","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:45864)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038963474Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"io task","client":"0","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"ds_connection connected from Ok(127.0.0.1:56524)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.038992139Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"io task","client":"2","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.039648806Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"client":"1","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.039668382Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"client":"2","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.039696667Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"client":"0","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.039989142Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040025075Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"[0]R flush_numbers[0..12]: [209, 216, 204, 212, 212, 202, 199, 216, 200, 203, 216, 212]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040128656Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"downstairs","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"[0]R generation[0..12]: [121, 124, 119, 123, 123, 118, 115, 124, 116, 118, 124, 123]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040140524Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"downstairs","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040151491Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"downstairs","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"[1]R flush_numbers[0..12]: [209, 216, 204, 212, 212, 202, 199, 216, 200, 203, 216, 212]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040162129Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"downstairs","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"[1]R generation[0..12]: [121, 124, 119, 123, 123, 118, 115, 124, 116, 118, 124, 123]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040172637Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"downstairs","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040183785Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"downstairs","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"[2]R flush_numbers[0..12]: [209, 216, 204, 212, 212, 202, 199, 216, 200, 203, 216, 212]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040194373Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"downstairs","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"[2]R generation[0..12]: [121, 124, 119, 123, 123, 118, 115, 124, 116, 118, 124, 123]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040204971Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"downstairs","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040217959Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"downstairs","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"Max found gen is 126","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040227267Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"downstairs","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"Generation requested: 126 >= found:126","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040236905Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"downstairs","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"Next flush: 218","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040246114Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"downstairs","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040255362Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"":"downstairs","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.04026496Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040274058Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"client":"0","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040284007Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"client":"1","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040293815Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"client":"2","":"downstairs","session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040303763Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040312931Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"e2ca5540-7b86-4a33-9ade-6164334af80c is now active with session: 02e9353e-0c4a-405e-9dbb-51a47ef604bd","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040322709Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040331378Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"session_id":"02e9353e-0c4a-405e-9dbb-51a47ef604bd"} {"msg":"The guest has finished waiting for activation with:126","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040340516Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637} {"msg":"Activated sub_volume e2ca5540-7b86-4a33-9ade-6164334af80c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040348505Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040356373Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.040650278Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"task":"crutest"} 01/30 Flush 02/30 Write block 183 len 28672 data: 11 10 10 11 8 8 10 03/30 Read block 193 len 16384 04/30 Read block 458 len 36864 05/30 Write block 239 len 8192 data: 6 9 06/30 Read block 574 len 32768 07/30 Flush 08/30 Write block 229 len 40960 data: 8 11 9 11 12 12 10 11 11 11 09/30 Write block 348 len 8192 data: 5 8 10/30 Read block 147 len 16384 11/30 Flush 12/30 Write block 432 len 40960 data: 4 5 6 8 6 7 6 7 8 9 13/30 Write block 440 len 16384 data: 9 10 11 11 14/30 Read block 331 len 12288 15/30 Read block 482 len 36864 16/30 Read block 503 len 28672 17/30 Write block 258 len 4096 data: 15 18/30 Write block 223 len 32768 data: 4 5 5 4 6 7 9 12 19/30 Write block 269 len 36864 data: 9 8 6 4 4 7 8 9 9 20/30 Read block 265 len 40960 21/30 Read block 579 len 24576 22/30 Write block 256 len 4096 data: 11 23/30 Write block 555 len 32768 data: 9 9 12 10 11 10 10 11 24/30 Write block 354 len 32768 data: 9 7 9 11 7 9 8 3 25/30 Read block 273 len 12288 26/30 Write block 33 len 32768 data: 14 11 10 8 9 12 12 12 27/30 Write block 223 len 8192 data: 5 6 28/30 Read block 14 len 24576 29/30 Read block 301 len 24576 30/30 Read block 72 len 4096 ---------------------------------------------------------------- Crucible gen:126 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 7 Done Done Done false 1002 Acked Read 4 Done Done Done false 1003 Acked Read 9 Done Done Done false 1004 Acked Write 2 Done Done Done false 1005 Acked Read 8 Done Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Write 10 Sent Done Done false 1008 Acked Write 2 Sent Done Done false 1009 Acked Read 4 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 10 Sent Done Done false 1012 Acked Write 4 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Read 9 Sent Done Done false 1015 Acked Read 7 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Read 6 Sent Done Done false 1021 Acked Write 1 Sent Done Done false 1022 Acked Write 8 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Write 8 Sent Done Done false 1026 Acked Write 2 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Read 1 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 0 0 24 Done 5 29 29 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1000)) Some(JobId(1010)) Some(JobId(1010)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:47.050020833Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:47.050205489Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1637,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 209 209 209 F F F 1 020-039 124 124 124 216 216 216 F T T <--- 2 040-059 119 119 119 204 204 204 F T T <--- 3 060-079 123 123 123 212 212 212 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 115 115 115 199 199 199 F F F 7 140-159 124 124 124 216 216 216 F F F 8 160-179 116 116 116 200 200 200 F F F 9 180-199 118 126 126 203 219 219 T F F <--- 10 200-219 124 124 124 216 216 216 F F F 11 220-239 123 126 126 212 220 220 T T T <--- 12 240-259 124 126 126 216 219 219 T T T <--- 13 260-279 125 125 125 217 217 217 F T T <--- 14 280-299 124 124 124 216 216 216 F F F 15 300-319 124 124 124 216 216 216 F F F 16 320-339 125 125 125 217 217 217 F F F 17 340-359 124 126 126 216 220 220 F T T <--- 18 360-379 116 116 116 200 200 200 F T T <--- 19 380-399 111 111 111 192 192 192 F F F 20 400-419 118 118 118 203 203 203 F F F 21 420-439 115 115 115 199 199 199 F T T <--- 22 440-459 122 122 122 211 211 211 F T T <--- 23 460-479 125 125 125 217 217 217 F F F 24 480-499 120 120 120 207 207 207 F F F 25 500-519 120 120 120 207 207 207 F F F 26 520-539 120 120 120 206 206 206 F F F 27 540-559 124 124 124 216 216 216 F T T <--- 28 560-579 124 124 124 213 213 213 F T T <--- 29 580-599 120 120 120 206 206 206 F F F Max gen: 126, Max flush: 220 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 63 Verifying data now No change after verify Loop: 63 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 209 209 209 F F F 1 020-039 127 127 127 221 221 221 F F F 2 040-059 127 127 127 221 221 221 F F F 3 060-079 123 123 123 212 212 212 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 115 115 115 199 199 199 F F F 7 140-159 124 124 124 216 216 216 F F F 8 160-179 116 116 116 200 200 200 F F F 9 180-199 126 126 126 219 219 219 F F F 10 200-219 124 124 124 216 216 216 F F F 11 220-239 127 127 127 221 221 221 F F F 12 240-259 127 127 127 221 221 221 F F F 13 260-279 127 127 127 221 221 221 F F F 14 280-299 124 124 124 216 216 216 F F F 15 300-319 124 124 124 216 216 216 F F F 16 320-339 125 125 125 217 217 217 F F F 17 340-359 127 127 127 221 221 221 F F F 18 360-379 127 127 127 221 221 221 F F F 19 380-399 111 111 111 192 192 192 F F F 20 400-419 118 118 118 203 203 203 F F F 21 420-439 127 127 127 221 221 221 F F F 22 440-459 127 127 127 221 221 221 F F F 23 460-479 125 125 125 217 217 217 F F F 24 480-499 120 120 120 207 207 207 F F F 25 500-519 120 120 120 207 207 207 F F F 26 520-539 120 120 120 206 206 206 F F F 27 540-559 127 127 127 221 221 221 F F F 28 560-579 127 127 127 221 221 221 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 127, Max flush: 221 Begin loop 64 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.310607812Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.318862446Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319151902Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319185905Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319195304Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319207091Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645} {"msg":"Crucible aef4e2cc-ce9b-47c7-a8d7-cb511d43e77c has session id: e24ba9a9-9bfe-455b-aa66-2745761bb138","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319541869Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"Upstairs opts: Upstairs UUID: aef4e2cc-ce9b-47c7-a8d7-cb511d43e77c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319570763Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"Crucible stats registered with UUID: aef4e2cc-ce9b-47c7-a8d7-cb511d43e77c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319581871Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:51.31959108Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319730503Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"io task","client":"1","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.31974712Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"io task","client":"0","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319756999Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"io task","client":"2","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319877126Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"task":"crutest"} {"msg":"The guest has requested activation with gen:128","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319901311Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645} {"msg":"aef4e2cc-ce9b-47c7-a8d7-cb511d43e77c active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319927667Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319936945Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"client":"0","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319946543Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"client":"1","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319956151Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"client":"2","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319965589Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"io task","client":"2","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319975488Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"io task","client":"0","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.319985416Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"io task","client":"1","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"ds_connection connected from Ok(127.0.0.1:55205)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.320073129Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"io task","client":"1","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"ds_connection connected from Ok(127.0.0.1:53409)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.320142086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"io task","client":"2","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"ds_connection connected from Ok(127.0.0.1:46734)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.320154524Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"io task","client":"0","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.320752162Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"client":"0","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.320782326Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"client":"1","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.320793564Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"client":"2","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321050556Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.32108399Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"[0]R flush_numbers[0..12]: [209, 221, 221, 212, 212, 202, 199, 216, 200, 219, 216, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.32113789Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"downstairs","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"[0]R generation[0..12]: [121, 127, 127, 123, 123, 118, 115, 124, 116, 126, 124, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321149058Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"downstairs","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321159986Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"downstairs","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"[1]R flush_numbers[0..12]: [209, 221, 221, 212, 212, 202, 199, 216, 200, 219, 216, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321170554Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"downstairs","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"[1]R generation[0..12]: [121, 127, 127, 123, 123, 118, 115, 124, 116, 126, 124, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321180872Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"downstairs","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.32119162Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"downstairs","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"[2]R flush_numbers[0..12]: [209, 221, 221, 212, 212, 202, 199, 216, 200, 219, 216, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321202148Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"downstairs","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"[2]R generation[0..12]: [121, 127, 127, 123, 123, 118, 115, 124, 116, 126, 124, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321212626Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"downstairs","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321223414Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"downstairs","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"Max found gen is 128","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321232762Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"downstairs","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"Generation requested: 128 >= found:128","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.32124261Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"downstairs","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"Next flush: 222","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321251829Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"downstairs","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321261197Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"":"downstairs","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321272805Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321281543Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"client":"0","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321291141Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"client":"1","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321300589Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"client":"2","":"downstairs","session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321310248Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321319256Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"aef4e2cc-ce9b-47c7-a8d7-cb511d43e77c is now active with session: e24ba9a9-9bfe-455b-aa66-2745761bb138","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321329134Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321337782Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"session_id":"e24ba9a9-9bfe-455b-aa66-2745761bb138"} {"msg":"The guest has finished waiting for activation with:128","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321346891Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645} {"msg":"Activated sub_volume aef4e2cc-ce9b-47c7-a8d7-cb511d43e77c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321354819Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321362848Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.321707033Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"task":"crutest"} 01/30 Write block 132 len 36864 data: 15 16 15 13 11 13 16 18 17 02/30 Write block 410 len 28672 data: 6 8 7 6 9 9 11 03/30 Write block 413 len 4096 data: 7 04/30 Write block 393 len 20480 data: 12 11 7 10 13 05/30 Read block 463 len 16384 06/30 Write block 41 len 20480 data: 14 16 17 15 16 07/30 Write block 550 len 20480 data: 9 10 11 10 10 08/30 Write block 310 len 36864 data: 14 14 11 11 12 12 13 14 11 09/30 Read block 325 len 16384 10/30 Read block 162 len 12288 11/30 Flush 12/30 Write block 256 len 32768 data: 12 12 16 14 18 18 15 13 13/30 Read block 175 len 8192 14/30 Flush 15/30 Write block 274 len 20480 data: 8 9 10 10 7 16/30 Write block 503 len 8192 data: 11 12 17/30 Flush 18/30 Write block 55 len 32768 data: 9 12 11 11 13 13 13 14 19/30 Read block 137 len 8192 20/30 Write block 209 len 16384 data: 12 14 11 13 21/30 Read block 535 len 20480 22/30 Write block 570 len 16384 data: 6 6 5 6 23/30 Read block 287 len 28672 24/30 Write block 433 len 20480 data: 6 7 9 7 8 25/30 Write block 318 len 24576 data: 12 11 11 11 7 8 26/30 Read block 121 len 24576 27/30 Read block 537 len 8192 28/30 Read block 562 len 36864 29/30 Write block 245 len 28672 data: 12 12 13 12 13 11 11 30/30 Read block 329 len 36864 ---------------------------------------------------------------- Crucible gen:128 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Write 7 Done Done Sent false 1002 Acked Write 1 Done Done Sent false 1003 Acked Write 5 Done Done Sent false 1004 Acked Read 4 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Write 5 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Read 4 Done Done Sent false 1009 Acked Read 3 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 8 Done Done Sent false 1012 Acked Read 2 Done Done Sent false 1013 Acked Flush 0 Done Done Sent false 1014 Acked Write 5 Done Done Sent false 1015 Acked Write 2 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 8 Done Done Sent false 1018 Acked Read 2 Done Done Sent false 1019 Acked Write 4 Done Done Sent false 1020 Acked Read 5 Done Done Sent false 1021 Acked Write 4 Done Done Sent false 1022 Acked Read 7 Done Done Sent false 1023 Acked Write 5 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Read 6 Done Done Sent false 1026 Acked Read 2 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Write 7 Done Done Sent false 1029 Acked Read 9 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 29 29 Done 30 30 1 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1016)) Some(JobId(1016)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:51.329980164Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:51.330168429Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1645,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 209 209 209 F F F 1 020-039 127 127 127 221 221 221 F F F 2 040-059 128 128 127 222 222 221 T T F <--- 3 060-079 123 123 123 212 212 212 T T F <--- 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 128 128 115 222 222 199 F F T <--- 7 140-159 128 128 124 222 222 216 F F T <--- 8 160-179 116 116 116 200 200 200 F F F 9 180-199 126 126 126 219 219 219 F F F 10 200-219 124 124 124 216 216 216 T T F <--- 11 220-239 127 127 127 221 221 221 F F F 12 240-259 128 128 127 223 223 221 T T F <--- 13 260-279 128 128 127 224 224 221 F F F <--- 14 280-299 124 124 124 216 216 216 F F F 15 300-319 128 128 124 222 222 216 T T F <--- 16 320-339 125 125 125 217 217 217 T T F <--- 17 340-359 127 127 127 221 221 221 F F F 18 360-379 127 127 127 221 221 221 F F F 19 380-399 128 128 111 222 222 192 F F F <--- 20 400-419 128 128 118 222 222 203 F F F <--- 21 420-439 127 127 127 221 221 221 T T F <--- 22 440-459 127 127 127 221 221 221 F F F 23 460-479 125 125 125 217 217 217 F F F 24 480-499 120 120 120 207 207 207 F F F 25 500-519 128 128 120 224 224 207 F F F <--- 26 520-539 120 120 120 206 206 206 F F F 27 540-559 128 128 127 222 222 221 F F F <--- 28 560-579 127 127 127 221 221 221 T T F <--- 29 580-599 120 120 120 206 206 206 F F F Max gen: 128, Max flush: 224 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 64 Verifying data now No change after verify Loop: 64 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 209 209 209 F F F 1 020-039 127 127 127 221 221 221 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 129 129 129 225 225 225 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 128 128 128 222 222 222 F F F 7 140-159 128 128 128 222 222 222 F F F 8 160-179 116 116 116 200 200 200 F F F 9 180-199 126 126 126 219 219 219 F F F 10 200-219 129 129 129 225 225 225 F F F 11 220-239 127 127 127 221 221 221 F F F 12 240-259 129 129 129 225 225 225 F F F 13 260-279 128 128 128 224 224 224 F F F 14 280-299 124 124 124 216 216 216 F F F 15 300-319 129 129 129 225 225 225 F F F 16 320-339 129 129 129 225 225 225 F F F 17 340-359 127 127 127 221 221 221 F F F 18 360-379 127 127 127 221 221 221 F F F 19 380-399 128 128 128 222 222 222 F F F 20 400-419 128 128 128 222 222 222 F F F 21 420-439 129 129 129 225 225 225 F F F 22 440-459 127 127 127 221 221 221 F F F 23 460-479 125 125 125 217 217 217 F F F 24 480-499 120 120 120 207 207 207 F F F 25 500-519 128 128 128 224 224 224 F F F 26 520-539 120 120 120 206 206 206 F F F 27 540-559 128 128 128 222 222 222 F F F 28 560-579 129 129 129 225 225 225 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 129, Max flush: 225 Begin loop 65 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.586711195Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.594840112Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595161522Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595196356Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595208324Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595220001Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653} {"msg":"Crucible 42aa6806-59af-4a38-8af6-ad594a0006b7 has session id: 8f54239c-9ce3-4476-9dbe-bc55e59ccb02","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595485932Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"Upstairs opts: Upstairs UUID: 42aa6806-59af-4a38-8af6-ad594a0006b7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595564087Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"Crucible stats registered with UUID: 42aa6806-59af-4a38-8af6-ad594a0006b7","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595589792Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:55.59559974Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595722777Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"io task","client":"0","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595739354Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"io task","client":"1","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595749372Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"io task","client":"2","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595896745Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"task":"crutest"} {"msg":"The guest has requested activation with gen:130","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.59592081Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653} {"msg":"42aa6806-59af-4a38-8af6-ad594a0006b7 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595933258Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595942086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"client":"0","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595951604Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"client":"1","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595962962Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"client":"2","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.59597258Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"io task","client":"2","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595982339Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"io task","client":"0","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.595992087Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"io task","client":"1","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"ds_connection connected from Ok(127.0.0.1:64058)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.596036348Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"io task","client":"2","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"ds_connection connected from Ok(127.0.0.1:64289)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.596047356Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"io task","client":"1","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"ds_connection connected from Ok(127.0.0.1:39577)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.596057634Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"io task","client":"0","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.596760563Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"client":"1","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.596793367Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"client":"0","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.59693834Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"client":"2","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597118436Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597141541Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"[0]R flush_numbers[0..12]: [209, 221, 225, 225, 212, 202, 222, 222, 200, 219, 225, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597229715Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"downstairs","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"[0]R generation[0..12]: [121, 127, 129, 129, 123, 118, 128, 128, 116, 126, 129, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597241733Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"downstairs","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597252861Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"downstairs","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"[1]R flush_numbers[0..12]: [209, 221, 225, 225, 212, 202, 222, 222, 200, 219, 225, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597265488Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"downstairs","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"[1]R generation[0..12]: [121, 127, 129, 129, 123, 118, 128, 128, 116, 126, 129, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597275936Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"downstairs","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597286724Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"downstairs","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"[2]R flush_numbers[0..12]: [209, 221, 225, 225, 212, 202, 222, 222, 200, 219, 225, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597297202Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"downstairs","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"[2]R generation[0..12]: [121, 127, 129, 129, 123, 118, 128, 128, 116, 126, 129, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.59730758Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"downstairs","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597318808Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"downstairs","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"Max found gen is 130","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597328326Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"downstairs","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"Generation requested: 130 >= found:130","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597337985Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"downstairs","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"Next flush: 226","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597347343Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"downstairs","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597356721Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"":"downstairs","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597366309Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597375028Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"client":"0","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597384666Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"client":"1","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597394214Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"client":"2","":"downstairs","session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597405372Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.59741442Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"42aa6806-59af-4a38-8af6-ad594a0006b7 is now active with session: 8f54239c-9ce3-4476-9dbe-bc55e59ccb02","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597424339Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"The guest has finished waiting for activation with:130","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597439286Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597446874Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"session_id":"8f54239c-9ce3-4476-9dbe-bc55e59ccb02"} {"msg":"Activated sub_volume 42aa6806-59af-4a38-8af6-ad594a0006b7","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597455933Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597464011Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"task":"crutest"} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.597694298Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"task":"crutest"} 01/30 Flush 02/30 Write block 25 len 16384 data: 11 11 9 12 03/30 Flush 04/30 Write block 510 len 8192 data: 6 8 05/30 Read block 551 len 12288 06/30 Flush 07/30 Write block 199 len 20480 data: 9 10 9 10 9 08/30 Write block 463 len 20480 data: 5 7 5 4 5 09/30 Write block 8 len 32768 data: 10 10 13 19 18 16 14 16 10/30 Read block 534 len 8192 11/30 Read block 83 len 8192 12/30 Write block 170 len 8192 data: 7 6 13/30 Write block 303 len 32768 data: 16 18 14 14 16 13 12 15 14/30 Write block 401 len 28672 data: 8 10 11 13 10 11 8 15/30 Read block 260 len 32768 16/30 Read block 415 len 12288 17/30 Write block 203 len 36864 data: 10 8 8 8 9 9 13 15 12 18/30 Write block 298 len 4096 data: 9 19/30 Read block 457 len 28672 20/30 Write block 174 len 12288 data: 11 8 8 21/30 Read block 41 len 32768 22/30 Read block 557 len 40960 23/30 Write block 251 len 32768 data: 12 12 11 10 12 13 13 17 24/30 Read block 447 len 32768 25/30 Write block 126 len 16384 data: 15 15 15 15 26/30 Read block 516 len 32768 27/30 Read block 476 len 36864 28/30 Write block 375 len 36864 data: 13 11 11 11 11 13 12 12 12 29/30 Read block 468 len 12288 30/30 Write block 502 len 8192 data: 10 12 ---------------------------------------------------------------- Crucible gen:130 GIO:true work queues: Upstairs:0 downstairs:27 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1003 Acked Write 2 Done Done Done false 1004 Acked Read 3 Done Done Done false 1005 Acked Flush 0 Done Done Sent false 1006 Acked Write 5 Done Done Sent false 1007 Acked Write 5 Done Done Sent false 1008 Acked Write 8 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Write 2 Done Done Sent false 1012 Acked Write 8 Done Done Sent false 1013 Acked Write 7 Done Done Sent false 1014 Acked Read 8 Done Done Sent false 1015 Acked Read 3 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Write 1 Done Done Sent false 1018 Acked Read 7 Done Done Sent false 1019 Acked Write 3 Done Done Sent false 1020 Acked Read 8 Done Done Sent false 1021 Acked Read 10 Done Done Sent false 1022 Acked Write 8 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Write 4 Done Done Sent false 1025 Acked Read 8 Done Done Sent false 1026 Acked Read 9 Done Done Sent false 1027 Acked Write 9 Done Done Sent false 1028 Acked Read 3 Done Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 2 25 28 Done 26 25 2 53 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1005)) Some(JobId(1005)) Some(JobId(1002)) Downstairs last five completed: 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:55.605831724Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:55.606033316Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1653,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 121 121 121 209 209 209 T T F <--- 1 020-039 130 130 130 227 227 227 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 129 129 129 225 225 225 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 128 128 128 222 222 222 T T F <--- 7 140-159 128 128 128 222 222 222 F F F 8 160-179 116 116 116 200 200 200 T T F <--- 9 180-199 126 126 126 219 219 219 T T F <--- 10 200-219 129 129 129 225 225 225 T T F <--- 11 220-239 127 127 127 221 221 221 F F F 12 240-259 129 129 129 225 225 225 T T F <--- 13 260-279 128 128 128 224 224 224 F F F 14 280-299 124 124 124 216 216 216 T T F <--- 15 300-319 129 129 129 225 225 225 T T F <--- 16 320-339 129 129 129 225 225 225 F F F 17 340-359 127 127 127 221 221 221 F F F 18 360-379 127 127 127 221 221 221 T T F <--- 19 380-399 128 128 128 222 222 222 T T F <--- 20 400-419 128 128 128 222 222 222 T T F <--- 21 420-439 129 129 129 225 225 225 F F F 22 440-459 127 127 127 221 221 221 F F F 23 460-479 125 125 125 217 217 217 T T F <--- 24 480-499 120 120 120 207 207 207 F F F 25 500-519 130 130 128 228 228 224 T T T <--- 26 520-539 120 120 120 206 206 206 F F F 27 540-559 128 128 128 222 222 222 F F F 28 560-579 129 129 129 225 225 225 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 130, Max flush: 228 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 65 Verifying data now No change after verify Loop: 65 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 131 131 131 229 229 229 F F F 1 020-039 130 130 130 227 227 227 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 129 129 129 225 225 225 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 131 131 131 229 229 229 F F F 7 140-159 128 128 128 222 222 222 F F F 8 160-179 131 131 131 229 229 229 F F F 9 180-199 131 131 131 229 229 229 F F F 10 200-219 131 131 131 229 229 229 F F F 11 220-239 127 127 127 221 221 221 F F F 12 240-259 131 131 131 229 229 229 F F F 13 260-279 128 128 128 224 224 224 F F F 14 280-299 131 131 131 229 229 229 F F F 15 300-319 131 131 131 229 229 229 F F F 16 320-339 129 129 129 225 225 225 F F F 17 340-359 127 127 127 221 221 221 F F F 18 360-379 131 131 131 229 229 229 F F F 19 380-399 131 131 131 229 229 229 F F F 20 400-419 131 131 131 229 229 229 F F F 21 420-439 129 129 129 225 225 225 F F F 22 440-459 127 127 127 221 221 221 F F F 23 460-479 131 131 131 229 229 229 F F F 24 480-499 120 120 120 207 207 207 F F F 25 500-519 131 131 131 229 229 229 F F F 26 520-539 120 120 120 206 206 206 F F F 27 540-559 128 128 128 222 222 222 F F F 28 560-579 129 129 129 225 225 225 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 131, Max flush: 229 Begin loop 66 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.862674964Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.870938586Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.871260066Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.871299548Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.871308737Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.871320254Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661} {"msg":"Crucible f06c9039-54f8-4edf-99a9-279feb422899 has session id: 00f5d700-6571-4072-a692-501e32ef2a91","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.871634126Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"Upstairs opts: Upstairs UUID: f06c9039-54f8-4edf-99a9-279feb422899, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.871658351Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"Crucible stats registered with UUID: f06c9039-54f8-4edf-99a9-279feb422899","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.871670709Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:59.871679627Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.871815492Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"io task","client":"1","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.871832668Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"io task","client":"0","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.871842527Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"io task","client":"2","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.871990819Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"task":"crutest"} {"msg":"The guest has requested activation with gen:132","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.872002687Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661} {"msg":"f06c9039-54f8-4edf-99a9-279feb422899 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.87203908Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.872048578Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"client":"0","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.872058156Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"client":"1","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.872067634Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"client":"2","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.872077073Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"io task","client":"2","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.872086961Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"io task","client":"0","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.872096819Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"io task","client":"1","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"ds_connection connected from Ok(127.0.0.1:39977)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.872109797Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"io task","client":"2","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"ds_connection connected from Ok(127.0.0.1:65082)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.872124904Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"io task","client":"0","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"ds_connection connected from Ok(127.0.0.1:42047)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.872137781Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"io task","client":"1","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.87289458Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"client":"1","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.872913806Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"client":"0","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.872924724Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"client":"2","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873229637Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873239245Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"[0]R flush_numbers[0..12]: [229, 227, 225, 225, 212, 202, 229, 222, 229, 229, 229, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873283467Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"downstairs","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"[0]R generation[0..12]: [131, 130, 129, 129, 123, 118, 131, 128, 131, 131, 131, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873294695Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"downstairs","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873305663Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"downstairs","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"[1]R flush_numbers[0..12]: [229, 227, 225, 225, 212, 202, 229, 222, 229, 229, 229, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873316161Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"downstairs","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"[1]R generation[0..12]: [131, 130, 129, 129, 123, 118, 131, 128, 131, 131, 131, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873326649Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"downstairs","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873337397Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"downstairs","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"[2]R flush_numbers[0..12]: [229, 227, 225, 225, 212, 202, 229, 222, 229, 229, 229, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873347945Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"downstairs","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"[2]R generation[0..12]: [131, 130, 129, 129, 123, 118, 131, 128, 131, 131, 131, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873360582Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"downstairs","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.8733713Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"downstairs","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"Max found gen is 132","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873381048Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"downstairs","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"Generation requested: 132 >= found:132","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873390687Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"downstairs","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"Next flush: 230","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873399955Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"downstairs","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873409333Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"":"downstairs","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873419291Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.87342808Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"client":"0","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873437798Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"client":"1","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873447436Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"client":"2","":"downstairs","session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873457214Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873466203Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"f06c9039-54f8-4edf-99a9-279feb422899 is now active with session: 00f5d700-6571-4072-a692-501e32ef2a91","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873476211Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873484749Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"session_id":"00f5d700-6571-4072-a692-501e32ef2a91"} {"msg":"The guest has finished waiting for activation with:132","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873493897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661} {"msg":"Activated sub_volume f06c9039-54f8-4edf-99a9-279feb422899","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873501856Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873510964Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.873843652Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"task":"crutest"} 01/30 Write block 123 len 4096 data: 13 02/30 Write block 334 len 16384 data: 14 14 14 13 03/30 Read block 144 len 4096 04/30 Write block 136 len 16384 data: 12 14 17 19 05/30 Read block 255 len 24576 06/30 Write block 471 len 4096 data: 9 07/30 Read block 264 len 40960 08/30 Write block 567 len 16384 data: 9 7 7 7 09/30 Write block 210 len 32768 data: 16 13 14 13 17 17 15 14 10/30 Write block 30 len 36864 data: 17 17 17 15 12 11 9 10 13 11/30 Read block 423 len 12288 12/30 Write block 289 len 40960 data: 11 11 10 11 10 9 7 8 8 10 13/30 Write block 306 len 36864 data: 15 17 14 13 16 15 12 12 13 14/30 Write block 358 len 24576 data: 8 10 9 4 4 4 15/30 Write block 135 len 40960 data: 14 13 15 18 20 18 17 14 11 11 16/30 Read block 385 len 32768 17/30 Write block 518 len 40960 data: 10 10 10 12 12 12 14 13 11 9 18/30 Read block 569 len 24576 19/30 Read block 333 len 4096 20/30 Read block 143 len 20480 21/30 Flush 22/30 Write block 420 len 8192 data: 8 9 23/30 Read block 285 len 12288 24/30 Flush 25/30 Write block 527 len 32768 data: 10 9 9 6 6 7 7 8 26/30 Write block 338 len 8192 data: 14 11 27/30 Read block 593 len 4096 28/30 Read block 243 len 24576 29/30 Read block 433 len 8192 30/30 Read block 588 len 32768 ---------------------------------------------------------------- Crucible gen:132 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 1 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Read 1 Done Done Done false 1003 Acked Write 4 Done Done Sent false 1004 Acked Read 6 Done Done Sent false 1005 Acked Write 1 Done Done Sent false 1006 Acked Read 10 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Write 8 Done Done Sent false 1009 Acked Write 9 Done Done Sent false 1010 Acked Read 3 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Write 9 Done Done Sent false 1013 Acked Write 6 Done Done Sent false 1014 Acked Write 10 Done Done Sent false 1015 Acked Read 8 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Read 6 Done Done Sent false 1018 Acked Read 1 Done Done Sent false 1019 Acked Read 5 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Read 3 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 8 Done Done Sent false 1025 Acked Write 2 Done Done Sent false 1026 Acked Read 1 Done Done Sent false 1027 Acked Read 6 Done Done Sent false 1028 Acked Read 2 Done Done Sent false 1029 Acked Read 8 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 27 27 Done 30 30 3 63 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:00:59.882085188Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:00:59.882268174Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1661,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 131 131 131 229 229 229 F F F 1 020-039 132 132 130 230 230 227 F F F <--- 2 040-059 129 129 129 225 225 225 F F F 3 060-079 129 129 129 225 225 225 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 132 132 131 230 230 229 F F T <--- 7 140-159 132 132 128 230 230 222 F F F <--- 8 160-179 131 131 131 229 229 229 F F F 9 180-199 131 131 131 229 229 229 F F F 10 200-219 132 132 131 230 230 229 F F F <--- 11 220-239 127 127 127 221 221 221 F F F 12 240-259 131 131 131 229 229 229 F F F 13 260-279 128 128 128 224 224 224 F F F 14 280-299 132 132 131 230 230 229 F F F <--- 15 300-319 132 132 131 230 230 229 F F F <--- 16 320-339 132 132 129 230 230 225 T T T <--- 17 340-359 132 132 127 230 230 221 F F F <--- 18 360-379 132 132 131 230 230 229 F F F <--- 19 380-399 131 131 131 229 229 229 F F F 20 400-419 131 131 131 229 229 229 F F F 21 420-439 132 132 129 231 231 225 F F F <--- 22 440-459 127 127 127 221 221 221 F F F 23 460-479 132 132 131 230 230 229 F F F <--- 24 480-499 120 120 120 207 207 207 F F F 25 500-519 132 132 131 230 230 229 F F F <--- 26 520-539 132 132 120 230 230 206 T T F <--- 27 540-559 128 128 128 222 222 222 F F F 28 560-579 132 132 129 230 230 225 F F F <--- 29 580-599 120 120 120 206 206 206 F F F Max gen: 132, Max flush: 231 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 66 Verifying data now No change after verify Loop: 66 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 131 131 131 229 229 229 F F F 1 020-039 132 132 132 230 230 230 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 129 129 129 225 225 225 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 132 132 132 230 230 230 F F F 7 140-159 132 132 132 230 230 230 F F F 8 160-179 131 131 131 229 229 229 F F F 9 180-199 131 131 131 229 229 229 F F F 10 200-219 132 132 132 230 230 230 F F F 11 220-239 127 127 127 221 221 221 F F F 12 240-259 131 131 131 229 229 229 F F F 13 260-279 128 128 128 224 224 224 F F F 14 280-299 132 132 132 230 230 230 F F F 15 300-319 132 132 132 230 230 230 F F F 16 320-339 133 133 133 232 232 232 F F F 17 340-359 132 132 132 230 230 230 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 131 131 131 229 229 229 F F F 20 400-419 131 131 131 229 229 229 F F F 21 420-439 132 132 132 231 231 231 F F F 22 440-459 127 127 127 221 221 221 F F F 23 460-479 132 132 132 230 230 230 F F F 24 480-499 120 120 120 207 207 207 F F F 25 500-519 132 132 132 230 230 230 F F F 26 520-539 133 133 133 232 232 232 F F F 27 540-559 128 128 128 222 222 222 F F F 28 560-579 132 132 132 230 230 230 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 133, Max flush: 232 Begin loop 67 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.136120904Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.144222947Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.14452438Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.144550296Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.144558754Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.144570122Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669} {"msg":"Crucible 83f318eb-7a81-4a66-acc1-94568b7d25eb has session id: fee7e7a0-a33a-459a-8161-9a5f89ae071e","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.144893531Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"Upstairs opts: Upstairs UUID: 83f318eb-7a81-4a66-acc1-94568b7d25eb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.144918217Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"Crucible stats registered with UUID: 83f318eb-7a81-4a66-acc1-94568b7d25eb","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.144927845Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:04.144966638Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.145122318Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"io task","client":"1","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.145154672Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"io task","client":"0","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.14516568Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"io task","client":"2","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.145285018Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"task":"crutest"} {"msg":"The guest has requested activation with gen:134","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.145304124Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669} {"msg":"83f318eb-7a81-4a66-acc1-94568b7d25eb active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.145330079Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.145339558Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"client":"0","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.145349146Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"client":"1","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.145358764Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"client":"2","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.145368362Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"io task","client":"2","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.14537813Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"io task","client":"0","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.145387919Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"io task","client":"1","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"ds_connection connected from Ok(127.0.0.1:36086)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.145443988Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"io task","client":"2","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"ds_connection connected from Ok(127.0.0.1:41611)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.145508546Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"io task","client":"1","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"ds_connection connected from Ok(127.0.0.1:48801)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.145552958Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"io task","client":"0","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146094896Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"client":"0","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146116172Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"client":"2","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146162794Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"client":"1","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146389621Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146400179Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"[0]R flush_numbers[0..12]: [229, 230, 225, 225, 212, 202, 230, 230, 229, 229, 230, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146478364Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"downstairs","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"[0]R generation[0..12]: [131, 132, 129, 129, 123, 118, 132, 132, 131, 131, 132, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146489502Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"downstairs","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.14650032Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"downstairs","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"[1]R flush_numbers[0..12]: [229, 230, 225, 225, 212, 202, 230, 230, 229, 229, 230, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146510948Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"downstairs","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"[1]R generation[0..12]: [131, 132, 129, 129, 123, 118, 132, 132, 131, 131, 132, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146521396Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"downstairs","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146532084Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"downstairs","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"[2]R flush_numbers[0..12]: [229, 230, 225, 225, 212, 202, 230, 230, 229, 229, 230, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146542532Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"downstairs","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"[2]R generation[0..12]: [131, 132, 129, 129, 123, 118, 132, 132, 131, 131, 132, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.14655287Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"downstairs","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146563668Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"downstairs","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"Max found gen is 134","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146573017Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"downstairs","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"Generation requested: 134 >= found:134","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146582585Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"downstairs","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"Next flush: 233","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146594123Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"downstairs","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146603391Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"":"downstairs","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146612999Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146621628Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"client":"0","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146631136Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"client":"1","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146640574Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"client":"2","":"downstairs","session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146650152Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146659141Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"83f318eb-7a81-4a66-acc1-94568b7d25eb is now active with session: fee7e7a0-a33a-459a-8161-9a5f89ae071e","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146669229Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146677747Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"session_id":"fee7e7a0-a33a-459a-8161-9a5f89ae071e"} {"msg":"The guest has finished waiting for activation with:134","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146686715Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669} {"msg":"Activated sub_volume 83f318eb-7a81-4a66-acc1-94568b7d25eb","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146694604Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.146702862Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.147026462Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"task":"crutest"} 01/30 Write block 23 len 8192 data: 10 10 02/30 Write block 327 len 20480 data: 11 8 9 11 11 03/30 Read block 287 len 4096 04/30 Read block 567 len 16384 05/30 Flush 06/30 Write block 573 len 8192 data: 7 6 07/30 Write block 337 len 12288 data: 14 15 12 08/30 Write block 127 len 20480 data: 16 16 16 18 15 09/30 Read block 304 len 20480 10/30 Read block 332 len 16384 11/30 Write block 520 len 4096 data: 11 12/30 Flush 13/30 Write block 490 len 20480 data: 10 10 9 8 8 14/30 Write block 252 len 12288 data: 13 12 11 15/30 Read block 427 len 36864 16/30 Write block 479 len 4096 data: 7 17/30 Write block 324 len 28672 data: 10 12 12 12 9 10 12 18/30 Write block 561 len 12288 data: 11 12 11 19/30 Read block 282 len 12288 20/30 Read block 273 len 36864 21/30 Read block 515 len 24576 22/30 Flush 23/30 Write block 159 len 12288 data: 7 4 6 24/30 Read block 518 len 20480 25/30 Write block 450 len 36864 data: 8 8 8 8 9 8 12 11 9 26/30 Read block 31 len 40960 27/30 Read block 516 len 28672 28/30 Write block 150 len 16384 data: 5 7 8 6 29/30 Write block 417 len 28672 data: 7 8 9 9 10 10 7 30/30 Read block 297 len 16384 ---------------------------------------------------------------- Crucible gen:134 GIO:true work queues: Upstairs:0 downstairs:25 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1005 Acked Write 2 Done Done Done false 1006 Acked Write 3 Done Done Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Read 5 Done Sent Done false 1009 Acked Read 4 Done Sent Done false 1010 Acked Write 1 Done Sent Done false 1011 Acked Flush 0 Done Sent Done false 1012 Acked Write 5 Done Sent Done false 1013 Acked Write 3 Done Sent Done false 1014 Acked Read 9 Done Sent Done false 1015 Acked Write 1 Done Sent Done false 1016 Acked Write 7 Done Sent Done false 1017 Acked Write 3 Done Sent Done false 1018 Acked Read 3 Done Sent Done false 1019 Acked Read 9 Done Sent Done false 1020 Acked Read 6 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 3 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Write 9 Done Sent Done false 1025 Acked Read 10 Done Sent Done false 1026 Acked Read 7 Done Sent Done false 1027 Acked Write 4 Done Sent Done false 1028 Acked Write 7 Done Sent Done false 1029 Acked Read 4 Done Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 23 0 23 Done 25 2 25 52 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1004)) Some(JobId(1021)) Downstairs last five completed: 1004 1003 1002 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:04.15515315Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:04.155349003Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1669,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 131 131 131 229 229 229 F F F 1 020-039 134 134 134 233 233 233 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 129 129 129 225 225 225 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 134 132 134 234 230 234 F F F <--- 7 140-159 132 132 132 230 230 230 T F T <--- 8 160-179 131 131 131 229 229 229 T F T <--- 9 180-199 131 131 131 229 229 229 F F F 10 200-219 132 132 132 230 230 230 F F F 11 220-239 127 127 127 221 221 221 F F F 12 240-259 134 131 134 235 229 235 F F F <--- 13 260-279 128 128 128 224 224 224 F F F 14 280-299 132 132 132 230 230 230 F F F 15 300-319 132 132 132 230 230 230 F F F 16 320-339 134 134 134 235 233 235 F T F <--- 17 340-359 132 132 132 230 230 230 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 131 131 131 229 229 229 F F F 20 400-419 131 131 131 229 229 229 T F T <--- 21 420-439 132 132 132 231 231 231 T F T <--- 22 440-459 127 127 127 221 221 221 T F T <--- 23 460-479 134 132 134 235 230 235 F F F <--- 24 480-499 134 120 134 235 207 235 F F F <--- 25 500-519 132 132 132 230 230 230 F F F 26 520-539 134 133 134 234 232 234 F F F <--- 27 540-559 128 128 128 222 222 222 F F F 28 560-579 134 132 134 235 230 235 F T F <--- 29 580-599 120 120 120 206 206 206 F F F Max gen: 134, Max flush: 235 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 67 Verifying data now No change after verify Loop: 67 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 131 131 131 229 229 229 F F F 1 020-039 134 134 134 233 233 233 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 129 129 129 225 225 225 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 135 135 135 236 236 236 F F F 8 160-179 135 135 135 236 236 236 F F F 9 180-199 131 131 131 229 229 229 F F F 10 200-219 132 132 132 230 230 230 F F F 11 220-239 127 127 127 221 221 221 F F F 12 240-259 134 134 134 235 235 235 F F F 13 260-279 128 128 128 224 224 224 F F F 14 280-299 132 132 132 230 230 230 F F F 15 300-319 132 132 132 230 230 230 F F F 16 320-339 134 134 134 235 235 235 F F F 17 340-359 132 132 132 230 230 230 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 131 131 131 229 229 229 F F F 20 400-419 135 135 135 236 236 236 F F F 21 420-439 135 135 135 236 236 236 F F F 22 440-459 135 135 135 236 236 236 F F F 23 460-479 134 134 134 235 235 235 F F F 24 480-499 134 134 134 235 235 235 F F F 25 500-519 132 132 132 230 230 230 F F F 26 520-539 134 134 134 234 234 234 F F F 27 540-559 128 128 128 222 222 222 F F F 28 560-579 134 134 134 235 235 235 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 135, Max flush: 236 Begin loop 68 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.407907785Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:08.415055097Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.415097039Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.421986139Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.42224781Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.422279004Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.422287442Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.42229885Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677} {"msg":"Crucible c3db5ddc-135d-4aa3-a718-e27fabfb6279 has session id: b32462ff-fab2-4130-9163-7a1160466860","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.422707114Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"Upstairs opts: Upstairs UUID: c3db5ddc-135d-4aa3-a718-e27fabfb6279, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.422731499Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"Crucible stats registered with UUID: c3db5ddc-135d-4aa3-a718-e27fabfb6279","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.422741097Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:08.422750766Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.422916995Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"io task","client":"2","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.422949758Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"io task","client":"0","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.422960936Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"io task","client":"1","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.423087283Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"task":"crutest"} {"msg":"The guest has requested activation with gen:136","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.42309924Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677} {"msg":"c3db5ddc-135d-4aa3-a718-e27fabfb6279 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.423115967Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.423127085Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"client":"0","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.423136923Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"client":"1","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.423146632Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"client":"2","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.42315619Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"io task","client":"2","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.423166108Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"io task","client":"0","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.423176106Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"io task","client":"1","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"ds_connection connected from Ok(127.0.0.1:39804)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.423190013Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"io task","client":"2","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"ds_connection connected from Ok(127.0.0.1:52082)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.42320927Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"io task","client":"1","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"ds_connection connected from Ok(127.0.0.1:50962)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.423221288Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"io task","client":"0","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.423907489Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"client":"1","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.423930425Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"client":"0","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.423941703Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"client":"2","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424212772Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424237597Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"[0]R flush_numbers[0..12]: [229, 233, 225, 225, 212, 202, 234, 236, 236, 229, 230, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424316742Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"downstairs","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"[0]R generation[0..12]: [131, 134, 129, 129, 123, 118, 134, 135, 135, 131, 132, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.42432991Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"downstairs","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424340998Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"downstairs","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"[1]R flush_numbers[0..12]: [229, 233, 225, 225, 212, 202, 234, 236, 236, 229, 230, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424351706Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"downstairs","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"[1]R generation[0..12]: [131, 134, 129, 129, 123, 118, 134, 135, 135, 131, 132, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424362224Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"downstairs","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424373002Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"downstairs","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"[2]R flush_numbers[0..12]: [229, 233, 225, 225, 212, 202, 234, 236, 236, 229, 230, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.42438367Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"downstairs","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"[2]R generation[0..12]: [131, 134, 129, 129, 123, 118, 134, 135, 135, 131, 132, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424394118Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"downstairs","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424404996Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"downstairs","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"Max found gen is 136","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424414454Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"downstairs","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"Generation requested: 136 >= found:136","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424424122Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"downstairs","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"Next flush: 237","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424433421Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"downstairs","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424442599Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"":"downstairs","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424452617Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424461385Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"client":"0","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424472743Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"client":"1","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424482521Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"client":"2","":"downstairs","session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424492639Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424501768Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"c3db5ddc-135d-4aa3-a718-e27fabfb6279 is now active with session: b32462ff-fab2-4130-9163-7a1160466860","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424511616Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424520284Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"session_id":"b32462ff-fab2-4130-9163-7a1160466860"} {"msg":"The guest has finished waiting for activation with:136","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424529343Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677} {"msg":"Activated sub_volume c3db5ddc-135d-4aa3-a718-e27fabfb6279","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.424537121Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.42454514Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.42486329Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"task":"crutest"} 01/30 Write block 158 len 40960 data: 7 8 5 7 6 6 8 9 9 8 02/30 Write block 185 len 12288 data: 11 12 9 03/30 Write block 488 len 8192 data: 7 10 04/30 Flush 05/30 Write block 458 len 28672 data: 10 10 9 5 5 6 8 06/30 Write block 355 len 12288 data: 8 10 12 07/30 Write block 12 len 8192 data: 19 17 08/30 Write block 171 len 20480 data: 7 9 9 12 9 09/30 Flush 10/30 Write block 64 len 20480 data: 10 11 12 12 12 11/30 Read block 156 len 32768 12/30 Read block 434 len 8192 13/30 Read block 308 len 20480 14/30 Flush 15/30 Write block 30 len 20480 data: 18 18 18 16 13 16/30 Read block 202 len 16384 17/30 Write block 410 len 40960 data: 7 9 8 8 10 10 12 8 9 10 18/30 Write block 569 len 12288 data: 8 8 7 19/30 Read block 576 len 8192 20/30 Flush 21/30 Write block 286 len 40960 data: 6 7 9 12 12 11 12 11 10 8 22/30 Write block 29 len 12288 data: 14 19 19 23/30 Read block 549 len 4096 24/30 Read block 550 len 36864 25/30 Write block 350 len 8192 data: 8 11 26/30 Read block 221 len 36864 27/30 Write block 22 len 16384 data: 10 11 11 12 28/30 Write block 151 len 24576 data: 8 9 7 6 7 8 29/30 Read block 104 len 40960 30/30 Write block 401 len 4096 data: 9 ---------------------------------------------------------------- Crucible gen:136 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Sent Done Done false 1001 Acked Write 3 Sent Done Done false 1002 Acked Write 2 Sent Done Done false 1003 Acked Flush 0 Sent Done Done false 1004 Acked Write 7 Sent Done Done false 1005 Acked Write 3 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Write 5 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 5 Sent Done Done false 1010 Acked Read 8 Sent Done Done false 1011 Acked Read 2 Sent Done Done false 1012 Acked Read 5 Sent Done Done false 1013 Acked Flush 0 Sent Done Done false 1014 Acked Write 5 Sent Done Done false 1015 Acked Read 4 Sent Done Done false 1016 Acked Write 10 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 10 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Read 1 Sent Done Done false 1023 Acked Read 9 Sent Done Done false 1024 Acked Write 2 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Write 4 Sent Done Done false 1027 Acked Write 6 Sent Done Done false 1028 Acked Read 10 Sent Done Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 2 33 Done 0 29 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:08.433331424Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:08.433535976Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1677,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 131 136 136 229 238 238 F F F <--- 1 020-039 134 136 136 233 240 240 F T T <--- 2 040-059 129 129 129 225 225 225 F F F 3 060-079 129 136 136 225 239 239 F F F <--- 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 135 136 136 236 237 237 F T T <--- 8 160-179 135 136 136 236 238 238 F F F <--- 9 180-199 131 136 136 229 237 237 F F F <--- 10 200-219 132 132 132 230 230 230 F F F 11 220-239 127 127 127 221 221 221 F F F 12 240-259 134 134 134 235 235 235 F F F 13 260-279 128 128 128 224 224 224 F F F 14 280-299 132 132 132 230 230 230 F T T <--- 15 300-319 132 132 132 230 230 230 F F F 16 320-339 134 134 134 235 235 235 F F F 17 340-359 132 136 136 230 238 238 F T T <--- 18 360-379 132 132 132 230 230 230 F F F 19 380-399 131 131 131 229 229 229 F F F 20 400-419 135 136 136 236 240 240 F T T <--- 21 420-439 135 135 135 236 236 236 F F F 22 440-459 135 136 136 236 238 238 F F F <--- 23 460-479 134 136 136 235 238 238 F F F <--- 24 480-499 134 136 136 235 237 237 F F F <--- 25 500-519 132 132 132 230 230 230 F F F 26 520-539 134 134 134 234 234 234 F F F 27 540-559 128 128 128 222 222 222 F F F 28 560-579 134 136 136 235 240 240 F F F <--- 29 580-599 120 120 120 206 206 206 F F F Max gen: 136, Max flush: 240 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 68 Verifying data now No change after verify Loop: 68 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 136 136 136 238 238 238 F F F 1 020-039 137 137 137 241 241 241 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 136 136 136 239 239 239 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 137 137 137 241 241 241 F F F 8 160-179 136 136 136 238 238 238 F F F 9 180-199 136 136 136 237 237 237 F F F 10 200-219 132 132 132 230 230 230 F F F 11 220-239 127 127 127 221 221 221 F F F 12 240-259 134 134 134 235 235 235 F F F 13 260-279 128 128 128 224 224 224 F F F 14 280-299 137 137 137 241 241 241 F F F 15 300-319 132 132 132 230 230 230 F F F 16 320-339 134 134 134 235 235 235 F F F 17 340-359 137 137 137 241 241 241 F F F 18 360-379 132 132 132 230 230 230 F F F 19 380-399 131 131 131 229 229 229 F F F 20 400-419 137 137 137 241 241 241 F F F 21 420-439 135 135 135 236 236 236 F F F 22 440-459 136 136 136 238 238 238 F F F 23 460-479 136 136 136 238 238 238 F F F 24 480-499 136 136 136 237 237 237 F F F 25 500-519 132 132 132 230 230 230 F F F 26 520-539 134 134 134 234 234 234 F F F 27 540-559 128 128 128 222 222 222 F F F 28 560-579 136 136 136 240 240 240 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 137, Max flush: 241 Begin loop 69 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.693695985Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:12.700829919Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.70087713Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.707633044Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.707921081Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.707956924Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.707968012Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.7079795Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685} {"msg":"Crucible 40383d8a-7d84-4d63-87ae-b29cb2232c4d has session id: c43f092c-a52a-4135-9f2f-d37afd652479","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708290211Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"Upstairs opts: Upstairs UUID: 40383d8a-7d84-4d63-87ae-b29cb2232c4d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708320376Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"Crucible stats registered with UUID: 40383d8a-7d84-4d63-87ae-b29cb2232c4d","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708331224Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:12.708340182Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708479726Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"io task","client":"0","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708496253Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"io task","client":"1","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708506161Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"io task","client":"2","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708656453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"task":"crutest"} {"msg":"The guest has requested activation with gen:138","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708680378Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685} {"msg":"40383d8a-7d84-4d63-87ae-b29cb2232c4d active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708697395Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708705964Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"client":"0","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708715462Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"client":"1","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.70872499Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"client":"2","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708736208Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"io task","client":"2","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708746176Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"io task","client":"0","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708756134Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"io task","client":"1","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"ds_connection connected from Ok(127.0.0.1:40177)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708804795Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"io task","client":"2","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"ds_connection connected from Ok(127.0.0.1:63466)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.70883029Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"io task","client":"1","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"ds_connection connected from Ok(127.0.0.1:58356)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.708861554Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"io task","client":"0","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709457703Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"client":"1","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709477449Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"client":"0","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709533868Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"client":"2","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709819495Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709829143Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"[0]R flush_numbers[0..12]: [238, 241, 225, 239, 212, 202, 234, 241, 238, 237, 230, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709870545Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"downstairs","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"[0]R generation[0..12]: [136, 137, 129, 136, 123, 118, 134, 137, 136, 136, 132, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709881853Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"downstairs","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709892721Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"downstairs","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"[1]R flush_numbers[0..12]: [238, 241, 225, 239, 212, 202, 234, 241, 238, 237, 230, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709905759Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"downstairs","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"[1]R generation[0..12]: [136, 137, 129, 136, 123, 118, 134, 137, 136, 136, 132, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709916227Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"downstairs","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709927115Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"downstairs","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"[2]R flush_numbers[0..12]: [238, 241, 225, 239, 212, 202, 234, 241, 238, 237, 230, 221]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709937633Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"downstairs","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"[2]R generation[0..12]: [136, 137, 129, 136, 123, 118, 134, 137, 136, 136, 132, 127]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709948011Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"downstairs","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709958859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"downstairs","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"Max found gen is 138","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709968187Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"downstairs","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"Generation requested: 138 >= found:138","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709977865Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"downstairs","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"Next flush: 242","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709987204Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"downstairs","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.709996462Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"":"downstairs","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.71000616Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.710015158Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"client":"0","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.710024837Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"client":"1","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.710034585Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"client":"2","":"downstairs","session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.710044263Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.710054641Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"40383d8a-7d84-4d63-87ae-b29cb2232c4d is now active with session: c43f092c-a52a-4135-9f2f-d37afd652479","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.710064489Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.710073247Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"session_id":"c43f092c-a52a-4135-9f2f-d37afd652479"} {"msg":"The guest has finished waiting for activation with:138","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.710082236Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685} {"msg":"Activated sub_volume 40383d8a-7d84-4d63-87ae-b29cb2232c4d","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.710090154Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :30,"time":"2026-07-20T23:01:12.710104062Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"task":"crutest"} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.710366503Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"task":"crutest"} 01/30 Write block 448 len 12288 data: 5 7 9 02/30 Write block 197 len 16384 data: 12 11 10 11 03/30 Write block 346 len 40960 data: 11 6 6 9 9 12 10 10 10 9 04/30 Read block 385 len 20480 05/30 Write block 243 len 40960 data: 11 10 13 13 14 13 14 12 13 14 06/30 Read block 23 len 32768 07/30 Write block 504 len 8192 data: 13 11 08/30 Write block 474 len 8192 data: 5 6 09/30 Flush 10/30 Write block 552 len 12288 data: 12 11 11 11/30 Write block 458 len 8192 data: 11 11 12/30 Write block 226 len 16384 data: 5 7 8 10 13/30 Write block 193 len 24576 data: 19 16 16 16 13 12 14/30 Read block 25 len 28672 15/30 Write block 436 len 8192 data: 8 9 16/30 Read block 454 len 40960 17/30 Read block 428 len 20480 18/30 Write block 469 len 32768 data: 7 9 10 7 6 6 7 7 19/30 Read block 119 len 20480 20/30 Read block 494 len 12288 21/30 Write block 430 len 16384 data: 8 5 5 7 22/30 Read block 43 len 12288 23/30 Flush 24/30 Write block 174 len 40960 data: 13 10 9 10 14 11 9 8 12 12 25/30 Read block 274 len 40960 26/30 Read block 161 len 12288 27/30 Read block 356 len 32768 28/30 Write block 371 len 16384 data: 10 10 9 11 29/30 Read block 133 len 20480 30/30 Write block 516 len 20480 data: 12 12 11 11 12 ---------------------------------------------------------------- Crucible gen:138 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 4 Done Done Done false 1002 Acked Write 10 Done Done Done false 1003 Acked Read 5 Sent Done Done false 1004 Acked Write 10 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Write 2 Sent Done Done false 1007 Acked Write 2 Sent Done Done false 1008 Acked Flush 0 Sent Done Done false 1009 Acked Write 3 Sent Done Done false 1010 Acked Write 2 Sent Done Done false 1011 Acked Write 4 Sent Done Done false 1012 Acked Write 6 Sent Done Done false 1013 Acked Read 7 Sent Done Done false 1014 Acked Write 2 Sent Done Done false 1015 Acked Read 10 Sent Done Done false 1016 Acked Read 5 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Read 5 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Write 4 Sent Done Done false 1021 Acked Read 3 Sent Done Done false 1022 Acked Flush 0 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Read 8 Sent Done Done false 1027 Acked Write 4 Sent Done Done false 1028 Acked Read 5 Sent Done Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 27 1 1 29 Done 3 29 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1022)) Some(JobId(1022)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:12.719050446Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:12.719220064Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1685,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 136 136 136 238 238 238 F F F 1 020-039 137 137 137 241 241 241 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 136 136 136 239 239 239 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 137 137 137 241 241 241 F F F 8 160-179 136 136 136 238 238 238 F T T <--- 9 180-199 136 138 138 237 243 243 T T T <--- 10 200-219 132 138 138 230 242 242 T F F <--- 11 220-239 127 138 138 221 243 243 F F F <--- 12 240-259 134 138 138 235 242 242 F F F <--- 13 260-279 128 128 128 224 224 224 F F F 14 280-299 137 137 137 241 241 241 F F F 15 300-319 132 132 132 230 230 230 F F F 16 320-339 134 134 134 235 235 235 F F F 17 340-359 137 138 138 241 242 242 T F F <--- 18 360-379 132 132 132 230 230 230 F T T <--- 19 380-399 131 131 131 229 229 229 F F F 20 400-419 137 137 137 241 241 241 F F F 21 420-439 135 138 138 236 243 243 F F F <--- 22 440-459 136 138 138 238 243 243 T F F <--- 23 460-479 136 138 138 238 243 243 F F F <--- 24 480-499 136 136 136 237 237 237 F F F 25 500-519 132 138 138 230 242 242 F T T <--- 26 520-539 134 134 134 234 234 234 F T T <--- 27 540-559 128 138 138 222 243 243 F F F <--- 28 560-579 136 136 136 240 240 240 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 138, Max flush: 243 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 69 Verifying data now No change after verify Loop: 69 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 136 136 136 238 238 238 F F F 1 020-039 137 137 137 241 241 241 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 136 136 136 239 239 239 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 137 137 137 241 241 241 F F F 8 160-179 139 139 139 244 244 244 F F F 9 180-199 139 139 139 244 244 244 F F F 10 200-219 138 138 138 242 242 242 F F F 11 220-239 138 138 138 243 243 243 F F F 12 240-259 138 138 138 242 242 242 F F F 13 260-279 128 128 128 224 224 224 F F F 14 280-299 137 137 137 241 241 241 F F F 15 300-319 132 132 132 230 230 230 F F F 16 320-339 134 134 134 235 235 235 F F F 17 340-359 138 138 138 242 242 242 F F F 18 360-379 139 139 139 244 244 244 F F F 19 380-399 131 131 131 229 229 229 F F F 20 400-419 137 137 137 241 241 241 F F F 21 420-439 138 138 138 243 243 243 F F F 22 440-459 138 138 138 243 243 243 F F F 23 460-479 138 138 138 243 243 243 F F F 24 480-499 136 136 136 237 237 237 F F F 25 500-519 139 139 139 244 244 244 F F F 26 520-539 139 139 139 244 244 244 F F F 27 540-559 138 138 138 243 243 243 F F F 28 560-579 136 136 136 240 240 240 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 139, Max flush: 244 Begin loop 70 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.983555461Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:16.990737586Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.990791266Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.997620707Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.997887257Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.997912902Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.99792149Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.997933118Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693} {"msg":"Crucible 08978739-aa76-47e8-9b8c-8eca278f4789 has session id: 2716971a-905b-40e5-9881-dad20064651a","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998279093Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"Upstairs opts: Upstairs UUID: 08978739-aa76-47e8-9b8c-8eca278f4789, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998307328Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"Crucible stats registered with UUID: 08978739-aa76-47e8-9b8c-8eca278f4789","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998318346Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:16.998327644Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998475827Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"io task","client":"1","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998505641Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"io task","client":"0","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998516579Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"io task","client":"2","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998626308Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"task":"crutest"} {"msg":"The guest has requested activation with gen:140","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998638156Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693} {"msg":"08978739-aa76-47e8-9b8c-8eca278f4789 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998661442Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.99867078Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"client":"0","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998680498Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"client":"1","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998690086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"client":"2","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998699615Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"io task","client":"2","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998731319Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"io task","client":"0","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998754354Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"io task","client":"1","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"ds_connection connected from Ok(127.0.0.1:38624)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998768012Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"io task","client":"0","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"ds_connection connected from Ok(127.0.0.1:38824)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.99877865Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"io task","client":"1","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"ds_connection connected from Ok(127.0.0.1:44854)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.998789098Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"io task","client":"2","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999410132Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"client":"2","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999439106Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"client":"1","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999450024Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"client":"0","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999715294Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999742919Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"[0]R flush_numbers[0..12]: [238, 241, 225, 239, 212, 202, 234, 241, 244, 244, 242, 243]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999798499Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"downstairs","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"[0]R generation[0..12]: [136, 137, 129, 136, 123, 118, 134, 137, 139, 139, 138, 138]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999809777Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"downstairs","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999820765Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"downstairs","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"[1]R flush_numbers[0..12]: [238, 241, 225, 239, 212, 202, 234, 241, 244, 244, 242, 243]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999831253Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"downstairs","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"[1]R generation[0..12]: [136, 137, 129, 136, 123, 118, 134, 137, 139, 139, 138, 138]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999841701Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"downstairs","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999852399Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"downstairs","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"[2]R flush_numbers[0..12]: [238, 241, 225, 239, 212, 202, 234, 241, 244, 244, 242, 243]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999864286Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"downstairs","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"[2]R generation[0..12]: [136, 137, 129, 136, 123, 118, 134, 137, 139, 139, 138, 138]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999874615Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"downstairs","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999885333Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"downstairs","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"Max found gen is 140","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999894561Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"downstairs","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"Generation requested: 140 >= found:140","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999904239Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"downstairs","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"Next flush: 245","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999913437Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"downstairs","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999922756Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"":"downstairs","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999932324Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999941032Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"client":"0","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.9999506Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"client":"1","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999960358Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"client":"2","":"downstairs","session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999970317Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999979405Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"08978739-aa76-47e8-9b8c-8eca278f4789 is now active with session: 2716971a-905b-40e5-9881-dad20064651a","v"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 :0,"name":"crucible","level":30,"time":"2026-07-20T23:01:16.999995042Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:17.00000393Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"session_id":"2716971a-905b-40e5-9881-dad20064651a"} {"msg":"The guest has finished waiting for activation with:140","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:17.000014628Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693} {"msg":"Activated sub_volume 08978739-aa76-47e8-9b8c-8eca278f4789","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:17.000022787Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:17.000030755Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"task":"crutest"} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:17.000470353Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"task":"crutest"} 01/30 Write block 334 len 28672 data: 15 15 15 15 16 13 11 02/30 Write block 222 len 20480 data: 7 6 7 6 6 03/30 Write block 312 len 12288 data: 13 13 14 04/30 Write block 66 len 12288 data: 13 13 13 05/30 Read block 57 len 24576 06/30 Write block 514 len 28672 data: 10 12 13 13 12 12 13 07/30 Write block 194 len 16384 data: 17 17 17 14 08/30 Flush 09/30 Write block 482 len 32768 data: 9 9 9 8 9 8 8 11 10/30 Read block 334 len 36864 11/30 Read block 404 len 36864 12/30 Read block 287 len 40960 13/30 Write block 527 len 8192 data: 11 10 14/30 Write block 279 len 12288 data: 9 8 8 15/30 Write block 323 len 4096 data: 9 16/30 Read block 123 len 24576 17/30 Write block 466 len 4096 data: 5 18/30 Read block 491 len 36864 19/30 Write block 410 len 40960 data: 8 10 9 9 11 11 13 9 10 11 20/30 Flush 21/30 Write block 493 len 28672 data: 9 9 8 6 7 9 10 22/30 Write block 319 len 36864 data: 12 12 12 8 10 11 13 13 13 23/30 Read block 263 len 40960 24/30 Write block 8 len 32768 data: 11 11 14 20 20 18 15 17 25/30 Read block 257 len 12288 26/30 Read block 95 len 12288 27/30 Read block 78 len 4096 28/30 Write block 331 len 40960 data: 12 14 12 16 16 16 16 17 14 12 29/30 Write block 555 len 24576 data: 10 10 13 11 12 11 30/30 Write block 65 len 12288 data: 12 14 14 ---------------------------------------------------------------- Crucible gen:140 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 5 Sent Done Done false 1002 Acked Write 3 Sent Done Done false 1003 Acked Write 3 Sent Done Done false 1004 Acked Read 6 Sent Done Done false 1005 Acked Write 7 Sent Done Done false 1006 Acked Write 4 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 8 Sent Done Done false 1009 Acked Read 9 Sent Done Done false 1010 Acked Read 9 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Write 2 Sent Done Done false 1013 Acked Write 3 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Read 6 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Read 9 Sent Done Done false 1018 Acked Write 10 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 7 Sent Done Done false 1021 Acked Write 9 Sent Done Done false 1022 Acked Read 10 Sent Done Done false 1023 Acked Write 8 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Read 3 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 6 Sent Sent Sent false 1029 Acked Write 3 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 3 3 35 Done 1 27 27 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1019)) Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:17.008810701Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:17.009017942Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1693,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 136 136 136 238 238 238 F T T <--- 1 020-039 137 137 137 241 241 241 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 136 140 140 239 245 245 F T T <--- 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 137 137 137 241 241 241 F F F 8 160-179 139 139 139 244 244 244 F F F 9 180-199 139 140 140 244 245 245 F F F <--- 10 200-219 138 138 138 242 242 242 F F F 11 220-239 138 140 140 243 245 245 F F F <--- 12 240-259 138 138 138 242 242 242 F F F 13 260-279 128 140 140 224 246 246 F F F <--- 14 280-299 137 140 140 241 246 246 F F F <--- 15 300-319 132 140 140 230 245 245 F T T <--- 16 320-339 134 140 140 235 246 246 T T T <--- 17 340-359 138 140 140 242 245 245 T T T <--- 18 360-379 139 139 139 244 244 244 F F F 19 380-399 131 131 131 229 229 229 F F F 20 400-419 137 140 140 241 246 246 F F F <--- 21 420-439 138 138 138 243 243 243 F F F 22 440-459 138 138 138 243 243 243 F F F 23 460-479 138 140 140 243 246 246 F F F <--- 24 480-499 136 140 140 237 246 246 F T T <--- 25 500-519 139 140 140 244 245 245 F F F <--- 26 520-539 139 140 140 244 246 246 F F F <--- 27 540-559 138 138 138 243 243 243 F T T <--- 28 560-579 136 136 136 240 240 240 F T T <--- 29 580-599 120 120 120 206 206 206 F F F Max gen: 140, Max flush: 246 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 70 Verifying data now No change after verify Loop: 70 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 247 247 247 F F F 1 020-039 137 137 137 241 241 241 F F F 2 040-059 129 129 129 225 225 225 F F F 3 060-079 141 141 141 247 247 247 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 137 137 137 241 241 241 F F F 8 160-179 139 139 139 244 244 244 F F F 9 180-199 140 140 140 245 245 245 F F F 10 200-219 138 138 138 242 242 242 F F F 11 220-239 140 140 140 245 245 245 F F F 12 240-259 138 138 138 242 242 242 F F F 13 260-279 140 140 140 246 246 246 F F F 14 280-299 140 140 140 246 246 246 F F F 15 300-319 141 141 141 247 247 247 F F F 16 320-339 141 141 141 247 247 247 F F F 17 340-359 141 141 141 247 247 247 F F F 18 360-379 139 139 139 244 244 244 F F F 19 380-399 131 131 131 229 229 229 F F F 20 400-419 140 140 140 246 246 246 F F F 21 420-439 138 138 138 243 243 243 F F F 22 440-459 138 138 138 243 243 243 F F F 23 460-479 140 140 140 246 246 246 F F F 24 480-499 141 141 141 247 247 247 F F F 25 500-519 140 140 140 245 245 245 F F F 26 520-539 140 140 140 246 246 246 F F F 27 540-559 141 141 141 247 247 247 F F F 28 560-579 141 141 141 247 247 247 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 141, Max flush: 247 Begin loop 71 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.276534223Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.284832409Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.285158268Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.285196461Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.285205239Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.285216777Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701} {"msg":"Crucible 6c92d873-6d82-4f01-9b43-7caca3395369 has session id: 2fb3ea68-07a3-4865-b3a3-eb292636e416","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.285606324Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"Upstairs opts: Upstairs UUID: 6c92d873-6d82-4f01-9b43-7caca3395369, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.285635019Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"Crucible stats registered with UUID: 6c92d873-6d82-4f01-9b43-7caca3395369","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.285646077Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:21.285657535Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.285801928Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"io task","client":"1","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.285833532Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"io task","client":"0","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.2858447Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"io task","client":"2","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.285969506Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"task":"crutest"} {"msg":"The guest has requested activation with gen:142","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.285985023Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701} {"msg":"6c92d873-6d82-4f01-9b43-7caca3395369 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.286008129Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.286017057Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"client":"0","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.286026726Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"client":"1","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.286036384Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"client":"2","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.286045842Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"io task","client":"2","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.28605573Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"io task","client":"0","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.286065758Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"io task","client":"1","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"ds_connection connected from Ok(127.0.0.1:40196)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.286137695Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"io task","client":"2","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"ds_connection connected from Ok(127.0.0.1:37562)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.286156261Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"io task","client":"0","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"ds_connection connected from Ok(127.0.0.1:55791)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.286235416Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"io task","client":"1","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.286824166Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"client":"1","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.28685683Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"client":"0","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.28696136Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"client":"2","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287140437Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287164982Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"[0]R flush_numbers[0..12]: [247, 241, 225, 247, 212, 202, 234, 241, 244, 245, 242, 245]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287237629Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"downstairs","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"[0]R generation[0..12]: [141, 137, 129, 141, 123, 118, 134, 137, 139, 140, 138, 140]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287248997Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"downstairs","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287259985Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"downstairs","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"[1]R flush_numbers[0..12]: [247, 241, 225, 247, 212, 202, 234, 241, 244, 245, 242, 245]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287270483Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"downstairs","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"[1]R generation[0..12]: [141, 137, 129, 141, 123, 118, 134, 137, 139, 140, 138, 140]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287280891Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"downstairs","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287291969Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"downstairs","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"[2]R flush_numbers[0..12]: [247, 241, 225, 247, 212, 202, 234, 241, 244, 245, 242, 245]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287302667Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"downstairs","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"[2]R generation[0..12]: [141, 137, 129, 141, 123, 118, 134, 137, 139, 140, 138, 140]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287313085Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"downstairs","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287324013Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"downstairs","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"Max found gen is 142","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.28733622Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"downstairs","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"Generation requested: 142 >= found:142","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287346048Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"downstairs","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"Next flush: 248","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287355297Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"downstairs","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287364675Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"":"downstairs","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287374483Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287383072Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"client":"0","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.28739264Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"client":"1","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287402158Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"client":"2","":"downstairs","session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287412406Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287421504Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"6c92d873-6d82-4f01-9b43-7caca3395369 is now active with session: 2fb3ea68-07a3-4865-b3a3-eb292636e416","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287431302Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287440001Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"session_id":"2fb3ea68-07a3-4865-b3a3-eb292636e416"} {"msg":"The guest has finished waiting for activation with:142","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287448999Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701} {"msg":"Activated sub_volume 6c92d873-6d82-4f01-9b43-7caca3395369","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287456958Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287465086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.287797374Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"task":"crutest"} 01/30 Write block 233 len 12288 data: 13 13 11 02/30 Write block 236 len 8192 data: 12 12 03/30 Write block 30 len 32768 data: 20 20 19 17 14 12 10 11 04/30 Read block 203 len 8192 05/30 Write block 387 len 36864 data: 7 8 6 8 10 11 13 12 8 06/30 Read block 533 len 36864 07/30 Write block 563 len 40960 data: 12 9 11 10 10 8 9 9 8 6 08/30 Write block 444 len 40960 data: 9 9 7 7 6 8 10 9 9 9 09/30 Read block 10 len 20480 10/30 Read block 287 len 16384 11/30 Read block 19 len 16384 12/30 Read block 366 len 4096 13/30 Write block 41 len 4096 data: 15 14/30 Write block 437 len 20480 data: 10 7 8 10 11 15/30 Read block 426 len 20480 16/30 Write block 435 len 4096 data: 10 17/30 Read block 415 len 24576 18/30 Read block 214 len 8192 19/30 Write block 291 len 40960 data: 12 13 12 11 9 9 9 11 10 12 20/30 Write block 209 len 28672 data: 14 17 14 15 14 18 18 21/30 Read block 526 len 4096 22/30 Read block 390 len 8192 23/30 Read block 255 len 12288 24/30 Read block 581 len 36864 25/30 Write block 497 len 40960 data: 8 10 11 11 9 11 13 14 12 9 26/30 Read block 179 len 12288 27/30 Write block 445 len 36864 data: 10 8 8 7 9 11 10 10 10 28/30 Read block 199 len 20480 29/30 Write block 330 len 12288 data: 13 13 15 30/30 Write block 53 len 32768 data: 4 7 10 13 12 12 14 14 ---------------------------------------------------------------- Crucible gen:142 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Write 8 Done Done Done false 1003 Acked Read 2 Done Done Done false 1004 Acked Write 9 Done Done Done false 1005 Acked Read 9 Done Done Done false 1006 Acked Write 10 Done Done Done false 1007 Acked Write 10 Done Done Done false 1008 Acked Read 5 Done Done Done false 1009 Acked Read 4 Done Done Done false 1010 Acked Read 4 Done Done Done false 1011 Acked Read 1 Done Done Done false 1012 Acked Write 1 Done Done Done false 1013 Acked Write 5 Done Done Done false 1014 Acked Read 5 Done Done Sent false 1015 Acked Write 1 Done Done Sent false 1016 Acked Read 6 Done Done Sent false 1017 Acked Read 2 Done Done Sent false 1018 Acked Write 10 Done Done Sent false 1019 Acked Write 7 Done Done Sent false 1020 Acked Read 1 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Read 3 Done Done Sent false 1023 Acked Read 9 Done Done Sent false 1024 Acked Write 10 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Write 9 Done Done Sent false 1027 Acked Read 5 Done Done Sent false 1028 Acked Write 3 Done Done Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 16 18 Done 29 29 14 72 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:21.296109987Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:21.296303721Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1701,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 247 247 247 F F F 1 020-039 137 137 137 241 241 241 T T T 2 040-059 129 129 129 225 225 225 T T T 3 060-079 141 141 141 247 247 247 T T F <--- 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 137 137 137 241 241 241 F F F 8 160-179 139 139 139 244 244 244 F F F 9 180-199 140 140 140 245 245 245 F F F 10 200-219 138 138 138 242 242 242 T T F <--- 11 220-239 140 140 140 245 245 245 T T T 12 240-259 138 138 138 242 242 242 F F F 13 260-279 140 140 140 246 246 246 F F F 14 280-299 140 140 140 246 246 246 T T F <--- 15 300-319 141 141 141 247 247 247 T T F <--- 16 320-339 141 141 141 247 247 247 T T F <--- 17 340-359 141 141 141 247 247 247 F F F 18 360-379 139 139 139 244 244 244 F F F 19 380-399 131 131 131 229 229 229 T T T 20 400-419 140 140 140 246 246 246 F F F 21 420-439 138 138 138 243 243 243 T T T 22 440-459 138 138 138 243 243 243 T T T 23 460-479 140 140 140 246 246 246 F F F 24 480-499 141 141 141 247 247 247 T T F <--- 25 500-519 140 140 140 245 245 245 T T F <--- 26 520-539 140 140 140 246 246 246 F F F 27 540-559 141 141 141 247 247 247 F F F 28 560-579 141 141 141 247 247 247 T T T 29 580-599 120 120 120 206 206 206 F F F Max gen: 141, Max flush: 247 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 71 Verifying data now No change after verify Loop: 71 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 141 141 141 247 247 247 F F F 1 020-039 142 142 142 248 248 248 F F F 2 040-059 142 142 142 248 248 248 F F F 3 060-079 142 142 142 248 248 248 F F F 4 080-099 123 123 123 212 212 212 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 137 137 137 241 241 241 F F F 8 160-179 139 139 139 244 244 244 F F F 9 180-199 140 140 140 245 245 245 F F F 10 200-219 142 142 142 248 248 248 F F F 11 220-239 142 142 142 248 248 248 F F F 12 240-259 138 138 138 242 242 242 F F F 13 260-279 140 140 140 246 246 246 F F F 14 280-299 142 142 142 248 248 248 F F F 15 300-319 142 142 142 248 248 248 F F F 16 320-339 142 142 142 248 248 248 F F F 17 340-359 141 141 141 247 247 247 F F F 18 360-379 139 139 139 244 244 244 F F F 19 380-399 142 142 142 248 248 248 F F F 20 400-419 140 140 140 246 246 246 F F F 21 420-439 142 142 142 248 248 248 F F F 22 440-459 142 142 142 248 248 248 F F F 23 460-479 140 140 140 246 246 246 F F F 24 480-499 142 142 142 248 248 248 F F F 25 500-519 142 142 142 248 248 248 F F F 26 520-539 140 140 140 246 246 246 F F F 27 540-559 141 141 141 247 247 247 F F F 28 560-579 142 142 142 248 248 248 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 142, Max flush: 248 Begin loop 72 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.558027267Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.566112493Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.566409327Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.566436062Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.566444631Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.566456099Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709} {"msg":"Crucible 19f2c45a-a717-489e-bf55-1449a8b3ac21 has session id: 1bc61236-191b-48ea-af41-48874a20563a","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.566863042Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"Upstairs opts: Upstairs UUID: 19f2c45a-a717-489e-bf55-1449a8b3ac21, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.566891477Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"Crucible stats registered with UUID: 19f2c45a-a717-489e-bf55-1449a8b3ac21","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.566902315Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:25.566911353Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567048628Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"io task","client":"1","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567079132Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"io task","client":"0","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.56709012Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"io task","client":"2","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567233223Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"task":"crutest"} {"msg":"The guest has requested activation with gen:144","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567252809Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709} {"msg":"19f2c45a-a717-489e-bf55-1449a8b3ac21 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567283184Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567297241Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"client":"0","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567306859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"client":"1","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567316357Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"client":"2","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567325876Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"io task","client":"2","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567335744Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"io task","client":"0","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567345652Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"io task","client":"1","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"ds_connection connected from Ok(127.0.0.1:50480)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567403731Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"io task","client":"1","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"ds_connection connected from Ok(127.0.0.1:56750)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567428167Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"io task","client":"0","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"ds_connection connected from Ok(127.0.0.1:43873)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.567439454Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"io task","client":"2","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.56810491Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"client":"0","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568127006Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"client":"1","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568138443Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"client":"2","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568418261Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568437927Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"[0]R flush_numbers[0..12]: [247, 248, 248, 248, 212, 202, 234, 241, 244, 245, 248, 248]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.56852837Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"downstairs","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"[0]R generation[0..12]: [141, 142, 142, 142, 123, 118, 134, 137, 139, 140, 142, 142]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568539808Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"downstairs","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568550826Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"downstairs","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"[1]R flush_numbers[0..12]: [247, 248, 248, 248, 212, 202, 234, 241, 244, 245, 248, 248]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568561484Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"downstairs","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"[1]R generation[0..12]: [141, 142, 142, 142, 123, 118, 134, 137, 139, 140, 142, 142]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568571952Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"downstairs","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.56858264Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"downstairs","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"[2]R flush_numbers[0..12]: [247, 248, 248, 248, 212, 202, 234, 241, 244, 245, 248, 248]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568593188Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"downstairs","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"[2]R generation[0..12]: [141, 142, 142, 142, 123, 118, 134, 137, 139, 140, 142, 142]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568603566Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"downstairs","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568614324Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"downstairs","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"Max found gen is 143","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568623583Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"downstairs","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"Generation requested: 144 >= found:143","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568633241Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"downstairs","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"Next flush: 249","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568642479Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"downstairs","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568651727Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"":"downstairs","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568661406Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568671234Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"client":"0","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568680902Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"client":"1","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.56869026Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"client":"2","":"downstairs","session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568699878Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568709017Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"19f2c45a-a717-489e-bf55-1449a8b3ac21 is now active with session: 1bc61236-191b-48ea-af41-48874a20563a","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568718825Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568727483Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"session_id":"1bc61236-191b-48ea-af41-48874a20563a"} {"msg":"The guest has finished waiting for activation with:144","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568736531Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709} {"msg":"Activated sub_volume 19f2c45a-a717-489e-bf55-1449a8b3ac21","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.56874442Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.568752339Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.569086246Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"task":"crutest"} 01/30 Write block 240 len 40960 data: 10 10 11 12 11 14 14 15 14 15 02/30 Flush 03/30 Write block 304 len 12288 data: 19 15 16 04/30 Read block 450 len 28672 05/30 Write block 19 len 16384 data: 13 12 9 11 06/30 Write block 85 len 24576 data: 12 11 7 9 11 9 07/30 Read block 554 len 36864 08/30 Read block 139 len 16384 09/30 Read block 442 len 12288 10/30 Read block 518 len 20480 11/30 Read block 137 len 8192 12/30 Read block 303 len 36864 13/30 Read block 294 len 20480 14/30 Read block 135 len 16384 15/30 Write block 70 len 8192 data: 7 7 16/30 Read block 104 len 36864 17/30 Read block 256 len 32768 18/30 Read block 580 len 28672 19/30 Flush 20/30 Write block 17 len 12288 data: 12 14 14 21/30 Read block 200 len 24576 22/30 Write block 329 len 8192 data: 11 14 23/30 Write block 404 len 8192 data: 14 11 24/30 Flush 25/30 Write block 393 len 20480 data: 14 13 9 11 14 26/30 Read block 559 len 12288 27/30 Read block 74 len 12288 28/30 Read block 191 len 36864 29/30 Write block 239 len 24576 data: 7 11 11 12 13 12 30/30 Read block 593 len 16384 ---------------------------------------------------------------- Crucible gen:144 GIO:true work queues: Upstairs:0 downstairs:28 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1002 Acked Write 3 Done Done Sent false 1003 Acked Read 7 Done Done Sent false 1004 Acked Write 4 Done Done Sent false 1005 Acked Write 6 Done Done Sent false 1006 Acked Read 9 Done Done Sent false 1007 Acked Read 4 Done Done Sent false 1008 Acked Read 3 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Read 2 Done Done Sent false 1011 Acked Read 9 Done Done Sent false 1012 Acked Read 5 Done Done Sent false 1013 Acked Read 4 Done Done Sent false 1014 Acked Write 2 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Read 8 Done Done Sent false 1017 Acked Read 7 Done Done Sent false 1018 Acked Flush 0 Done Done Sent false 1019 Acked Write 3 Done Done Sent false 1020 Acked Read 6 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Write 2 Done Done Sent false 1023 Acked Flush 0 Done Done Sent false 1024 Acked Write 5 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Read 3 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Write 6 Done Done Sent false 1029 Acked Read 4 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 0 28 29 Done 27 28 0 55 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1023)) Some(JobId(1001)) Downstairs last five completed: 1001 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:25.578020493Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:25.578223765Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1709,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 144 144 141 251 251 247 F F F <--- 1 020-039 144 144 142 250 250 248 F F F <--- 2 040-059 142 142 142 248 248 248 F F F 3 060-079 144 144 142 250 250 248 F F F <--- 4 080-099 144 144 123 250 250 212 F F F <--- 5 100-119 118 118 118 202 202 202 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 137 137 137 241 241 241 F F F 8 160-179 139 139 139 244 244 244 F F F 9 180-199 140 140 140 245 245 245 F F F 10 200-219 142 142 142 248 248 248 F F F 11 220-239 142 142 142 248 248 248 T T F <--- 12 240-259 144 144 144 249 249 249 T T F <--- 13 260-279 140 140 140 246 246 246 F F F 14 280-299 142 142 142 248 248 248 F F F 15 300-319 144 144 142 250 250 248 F F F <--- 16 320-339 144 144 142 251 251 248 F F F <--- 17 340-359 141 141 141 247 247 247 F F F 18 360-379 139 139 139 244 244 244 F F F 19 380-399 142 142 142 248 248 248 T T F <--- 20 400-419 144 144 140 251 251 246 F F F <--- 21 420-439 142 142 142 248 248 248 F F F 22 440-459 142 142 142 248 248 248 F F F 23 460-479 140 140 140 246 246 246 F F F 24 480-499 142 142 142 248 248 248 F F F 25 500-519 142 142 142 248 248 248 F F F 26 520-539 140 140 140 246 246 246 F F F 27 540-559 141 141 141 247 247 247 F F F 28 560-579 142 142 142 248 248 248 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 144, Max flush: 251 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 72 Verifying data now No change after verify Loop: 72 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 144 144 144 251 251 251 F F F 1 020-039 144 144 144 250 250 250 F F F 2 040-059 142 142 142 248 248 248 F F F 3 060-079 144 144 144 250 250 250 F F F 4 080-099 144 144 144 250 250 250 F F F 5 100-119 118 118 118 202 202 202 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 137 137 137 241 241 241 F F F 8 160-179 139 139 139 244 244 244 F F F 9 180-199 140 140 140 245 245 245 F F F 10 200-219 142 142 142 248 248 248 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 145 145 145 252 252 252 F F F 13 260-279 140 140 140 246 246 246 F F F 14 280-299 142 142 142 248 248 248 F F F 15 300-319 144 144 144 250 250 250 F F F 16 320-339 144 144 144 251 251 251 F F F 17 340-359 141 141 141 247 247 247 F F F 18 360-379 139 139 139 244 244 244 F F F 19 380-399 145 145 145 252 252 252 F F F 20 400-419 144 144 144 251 251 251 F F F 21 420-439 142 142 142 248 248 248 F F F 22 440-459 142 142 142 248 248 248 F F F 23 460-479 140 140 140 246 246 246 F F F 24 480-499 142 142 142 248 248 248 F F F 25 500-519 142 142 142 248 248 248 F F F 26 520-539 140 140 140 246 246 246 F F F 27 540-559 141 141 141 247 247 247 F F F 28 560-579 142 142 142 248 248 248 F F F 29 580-599 120 120 120 206 206 206 F F F Max gen: 145, Max flush: 252 Begin loop 73 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.8215823Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:29.828782672Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.828838191Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.835602175Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.835879493Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.835907817Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.835918805Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.835930263Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717} {"msg":"Crucible 8f5879a7-08cb-46f1-a918-b12f599ae94a has session id: 1ee13e54-de55-4604-bed6-062fa56bf00d","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836252653Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"Upstairs opts: Upstairs UUID: 8f5879a7-08cb-46f1-a918-b12f599ae94a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836282277Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"Crucible stats registered with UUID: 8f5879a7-08cb-46f1-a918-b12f599ae94a","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836293185Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:29.836302273Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836456295Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"io task","client":"0","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836486149Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"io task","client":"1","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836497107Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"io task","client":"2","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836656047Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"task":"crutest"} {"msg":"The guest has requested activation with gen:146","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836672094Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717} {"msg":"8f5879a7-08cb-46f1-a918-b12f599ae94a active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836706398Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836715836Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"client":"0","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836725604Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"client":"1","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836737382Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"client":"2","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836769556Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"io task","client":"2","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836792492Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"io task","client":"0","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.83680337Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"io task","client":"1","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"ds_connection connected from Ok(127.0.0.1:55569)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836814098Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"io task","client":"1","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"ds_connection connected from Ok(127.0.0.1:52634)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836824446Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"io task","client":"2","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"ds_connection connected from Ok(127.0.0.1:64941)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.836843122Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"io task","client":"0","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.83749355Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"client":"1","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.837524085Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"client":"2","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.837636174Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"client":"0","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.837792124Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.83781634Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"[0]R flush_numbers[0..12]: [251, 250, 248, 250, 250, 202, 234, 241, 244, 245, 248, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.837908193Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"downstairs","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"[0]R generation[0..12]: [144, 144, 142, 144, 144, 118, 134, 137, 139, 140, 142, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.837919361Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"downstairs","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.837930389Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"downstairs","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"[1]R flush_numbers[0..12]: [251, 250, 248, 250, 250, 202, 234, 241, 244, 245, 248, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.837943036Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"downstairs","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"[1]R generation[0..12]: [144, 144, 142, 144, 144, 118, 134, 137, 139, 140, 142, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.837953494Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"downstairs","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.837964172Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"downstairs","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"[2]R flush_numbers[0..12]: [251, 250, 248, 250, 250, 202, 234, 241, 244, 245, 248, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.83797456Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"downstairs","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"[2]R generation[0..12]: [144, 144, 142, 144, 144, 118, 134, 137, 139, 140, 142, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.837985028Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"downstairs","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.837995736Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"downstairs","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"Max found gen is 146","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838005175Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"downstairs","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"Generation requested: 146 >= found:146","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838015063Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"downstairs","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"Next flush: 253","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838024441Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"downstairs","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838033809Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"":"downstairs","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838043417Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838052226Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"client":"0","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838061894Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"client":"1","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838071452Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"client":"2","":"downstairs","session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.8380811Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838091638Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"8f5879a7-08cb-46f1-a918-b12f599ae94a is now active with session: 1ee13e54-de55-4604-bed6-062fa56bf00d","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838101466Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838110005Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"session_id":"1ee13e54-de55-4604-bed6-062fa56bf00d"} {"msg":"The guest has finished waiting for activation with:146","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838119013Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717} {"msg":"Activated sub_volume 8f5879a7-08cb-46f1-a918-b12f599ae94a","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838126872Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.83813489Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.838461219Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"task":"crutest"} 01/30 Write block 408 len 20480 data: 7 5 9 11 10 02/30 Write block 44 len 24576 data: 16 17 12 10 8 7 03/30 Write block 510 len 40960 data: 7 9 8 8 11 13 14 14 13 13 04/30 Write block 117 len 8192 data: 8 11 05/30 Read block 199 len 4096 06/30 Write block 47 len 40960 data: 11 9 8 6 6 5 5 8 11 14 07/30 Read block 279 len 8192 08/30 Read block 330 len 32768 09/30 Read block 405 len 4096 10/30 Read block 413 len 40960 11/30 Read block 73 len 40960 12/30 Read block 479 len 4096 13/30 Write block 478 len 12288 data: 7 8 5 14/30 Read block 528 len 12288 15/30 Read block 553 len 20480 16/30 Flush 17/30 Flush 18/30 Write block 26 len 32768 data: 12 10 13 15 21 21 20 18 19/30 Read block 441 len 8192 20/30 Write block 379 len 20480 data: 12 14 13 13 13 21/30 Write block 400 len 12288 data: 7 10 11 22/30 Write block 90 len 4096 data: 10 23/30 Write block 175 len 40960 data: 11 10 11 15 12 10 9 13 13 11 24/30 Write block 596 len 12288 data: 11 7 6 25/30 Read block 437 len 36864 26/30 Write block 418 len 24576 data: 11 12 10 11 11 8 27/30 Read block 421 len 40960 28/30 Read block 13 len 4096 29/30 Read block 441 len 28672 30/30 Read block 535 len 24576 ---------------------------------------------------------------- Crucible gen:146 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 5 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Write 10 Sent Done Done false 1003 Acked Write 2 Sent Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Write 10 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Read 1 Sent Done Done false 1009 Acked Read 10 Sent Done Done false 1010 Acked Read 10 Sent Done Done false 1011 Acked Read 1 Sent Done Done false 1012 Acked Write 3 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Read 5 Sent Done Done false 1015 Acked Flush 0 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Write 5 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Write 1 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 9 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Read 10 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Read 7 Sent Done Done false 1029 Acked Read 6 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 0 28 Done 2 30 30 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:29.846557003Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:29.846745148Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1717,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 144 144 144 251 251 251 F F F 1 020-039 144 144 144 250 250 250 F T T <--- 2 040-059 142 146 146 248 253 253 T F F <--- 3 060-079 144 144 144 250 250 250 F F F 4 080-099 144 144 144 250 250 250 F T T <--- 5 100-119 118 146 146 202 253 253 F F F <--- 6 120-139 134 134 134 234 234 234 F F F 7 140-159 137 137 137 241 241 241 F F F 8 160-179 139 139 139 244 244 244 F T T <--- 9 180-199 140 140 140 245 245 245 F T T <--- 10 200-219 142 142 142 248 248 248 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 145 145 145 252 252 252 F F F 13 260-279 140 140 140 246 246 246 F F F 14 280-299 142 142 142 248 248 248 F F F 15 300-319 144 144 144 250 250 250 F F F 16 320-339 144 144 144 251 251 251 F F F 17 340-359 141 141 141 247 247 247 F F F 18 360-379 139 139 139 244 244 244 F T T <--- 19 380-399 145 145 145 252 252 252 F T T <--- 20 400-419 144 146 146 251 253 253 T T T <--- 21 420-439 142 142 142 248 248 248 F T T <--- 22 440-459 142 142 142 248 248 248 F F F 23 460-479 140 146 146 246 253 253 F F F <--- 24 480-499 142 146 146 248 253 253 F F F <--- 25 500-519 142 146 146 248 253 253 F F F <--- 26 520-539 140 140 140 246 246 246 F F F 27 540-559 141 141 141 247 247 247 F F F 28 560-579 142 142 142 248 248 248 F F F 29 580-599 120 120 120 206 206 206 F T T <--- Max gen: 146, Max flush: 253 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 73 Verifying data now No change after verify Loop: 73 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 144 144 144 251 251 251 F F F 1 020-039 147 147 147 254 254 254 F F F 2 040-059 146 146 146 253 253 253 F F F 3 060-079 144 144 144 250 250 250 F F F 4 080-099 147 147 147 254 254 254 F F F 5 100-119 146 146 146 253 253 253 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 137 137 137 241 241 241 F F F 8 160-179 147 147 147 254 254 254 F F F 9 180-199 147 147 147 254 254 254 F F F 10 200-219 142 142 142 248 248 248 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 145 145 145 252 252 252 F F F 13 260-279 140 140 140 246 246 246 F F F 14 280-299 142 142 142 248 248 248 F F F 15 300-319 144 144 144 250 250 250 F F F 16 320-339 144 144 144 251 251 251 F F F 17 340-359 141 141 141 247 247 247 F F F 18 360-379 147 147 147 254 254 254 F F F 19 380-399 147 147 147 254 254 254 F F F 20 400-419 147 147 147 254 254 254 F F F 21 420-439 147 147 147 254 254 254 F F F 22 440-459 142 142 142 248 248 248 F F F 23 460-479 146 146 146 253 253 253 F F F 24 480-499 146 146 146 253 253 253 F F F 25 500-519 146 146 146 253 253 253 F F F 26 520-539 140 140 140 246 246 246 F F F 27 540-559 141 141 141 247 247 247 F F F 28 560-579 142 142 142 248 248 248 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 147, Max flush: 254 Begin loop 74 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.109859664Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.118197862Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.118497146Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.118533889Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.118542817Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.118554105Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725} {"msg":"Crucible 6cf44caa-5235-462f-9006-ff60beaf5a47 has session id: 671e02cc-e223-40ca-bb2b-e2dafbb4e480","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.118870836Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"Upstairs opts: Upstairs UUID: 6cf44caa-5235-462f-9006-ff60beaf5a47, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.1189029Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"Crucible stats registered with UUID: 6cf44caa-5235-462f-9006-ff60beaf5a47","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.118914388Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:34.118923616Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.119049802Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"io task","client":"1","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.119080267Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"io task","client":"0","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.119091265Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"io task","client":"2","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.11922365Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"task":"crutest"} {"msg":"The guest has requested activation with gen:148","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.119237937Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725} {"msg":"6cf44caa-5235-462f-9006-ff60beaf5a47 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.119255854Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.119264642Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"client":"0","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.11927408Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"client":"1","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.119283669Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"client":"2","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.119293247Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"io task","client":"2","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.119303035Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"io task","client":"0","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.119312883Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"io task","client":"1","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"ds_connection connected from Ok(127.0.0.1:39561)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.119379551Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"io task","client":"2","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"ds_connection connected from Ok(127.0.0.1:63196)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.119406516Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"io task","client":"0","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"ds_connection connected from Ok(127.0.0.1:45478)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.119417774Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"io task","client":"1","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120012882Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"client":"0","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120038067Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"client":"2","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120049825Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"client":"1","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120319285Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120347499Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"[0]R flush_numbers[0..12]: [251, 254, 253, 250, 254, 253, 234, 241, 254, 254, 248, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120403039Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"downstairs","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"[0]R generation[0..12]: [144, 147, 146, 144, 147, 146, 134, 137, 147, 147, 142, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120415017Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"downstairs","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120426095Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"downstairs","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"[1]R flush_numbers[0..12]: [251, 254, 253, 250, 254, 253, 234, 241, 254, 254, 248, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120436753Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"downstairs","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"[1]R generation[0..12]: [144, 147, 146, 144, 147, 146, 134, 137, 147, 147, 142, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120447231Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"downstairs","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120458209Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"downstairs","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"[2]R flush_numbers[0..12]: [251, 254, 253, 250, 254, 253, 234, 241, 254, 254, 248, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120468747Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"downstairs","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"[2]R generation[0..12]: [144, 147, 146, 144, 147, 146, 134, 137, 147, 147, 142, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120481854Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"downstairs","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120492872Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"downstairs","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"Max found gen is 148","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.12050238Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"downstairs","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"Generation requested: 148 >= found:148","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120512259Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"downstairs","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"Next flush: 255","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120521657Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"downstairs","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120531085Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"":"downstairs","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120540733Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120549372Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"client":"0","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.12055925Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"client":"1","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120568998Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"client":"2","":"downstairs","session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120578756Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120587864Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"6cf44caa-5235-462f-9006-ff60beaf5a47 is now active with session: 671e02cc-e223-40ca-bb2b-e2dafbb4e480","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120597763Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120606481Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"session_id":"671e02cc-e223-40ca-bb2b-e2dafbb4e480"} {"msg":"The guest has finished waiting for activation with:148","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120615479Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725} {"msg":"Activated sub_volume 6cf44caa-5235-462f-9006-ff60beaf5a47","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120623298Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120632566Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.120956785Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"task":"crutest"} 01/30 Write block 272 len 12288 data: 5 5 9 02/30 Read block 397 len 20480 03/30 Write block 1 len 4096 data: 4 04/30 Write block 55 len 12288 data: 12 15 13 05/30 Read block 75 len 36864 06/30 Write block 337 len 40960 data: 17 18 15 13 9 11 12 13 12 12 07/30 Read block 257 len 16384 08/30 Read block 57 len 24576 09/30 Write block 23 len 8192 data: 12 12 10/30 Read block 406 len 24576 11/30 Read block 385 len 16384 12/30 Write block 184 len 36864 data: 12 12 13 10 9 11 11 13 15 13/30 Write block 484 len 32768 data: 10 9 10 9 9 12 11 11 14/30 Write block 458 len 4096 data: 12 15/30 Write block 79 len 12288 data: 7 6 7 16/30 Write block 31 len 40960 data: 22 21 19 15 13 11 12 14 13 13 17/30 Read block 299 len 12288 18/30 Read block 61 len 28672 19/30 Read block 321 len 8192 20/30 Read block 552 len 24576 21/30 Read block 590 len 40960 22/30 Read block 45 len 8192 23/30 Read block 93 len 20480 24/30 Read block 420 len 32768 25/30 Read block 584 len 12288 26/30 Write block 535 len 12288 data: 7 6 6 27/30 Write block 149 len 8192 data: 6 6 28/30 Read block 147 len 36864 29/30 Write block 328 len 28672 data: 10 12 15 14 16 13 17 30/30 Write block 275 len 4096 data: 10 ---------------------------------------------------------------- Crucible gen:148 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 5 Done Done Done false 1002 Acked Write 1 Done Done Done false 1003 Acked Write 3 Done Done Done false 1004 Acked Read 9 Done Sent Done false 1005 Acked Write 10 Done Sent Done false 1006 Acked Read 4 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Write 2 Done Sent Done false 1009 Acked Read 6 Done Sent Done false 1010 Acked Read 4 Done Sent Done false 1011 Acked Write 9 Done Sent Done false 1012 Acked Write 8 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Write 3 Done Sent Done false 1015 Acked Write 10 Done Sent Done false 1016 Acked Read 3 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Read 2 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Read 10 Done Sent Done false 1021 Acked Read 2 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 8 Done Sent Done false 1024 Acked Read 3 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Read 9 Done Sent Done false 1028 Acked Write 7 Sent Sent Sent false 1029 Acked Write 1 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 26 2 30 Done 28 4 28 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:34.128324845Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:34.128535546Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1725,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 144 144 144 251 251 251 T T T 1 020-039 147 147 147 254 254 254 T F T <--- 2 040-059 146 146 146 253 253 253 T T T 3 060-079 144 144 144 250 250 250 T F T <--- 4 080-099 147 147 147 254 254 254 T F T <--- 5 100-119 146 146 146 253 253 253 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 137 137 137 241 241 241 T F T <--- 8 160-179 147 147 147 254 254 254 F F F 9 180-199 147 147 147 254 254 254 T F T <--- 10 200-219 142 142 142 248 248 248 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 145 145 145 252 252 252 F F F 13 260-279 140 140 140 246 246 246 T T T 14 280-299 142 142 142 248 248 248 F F F 15 300-319 144 144 144 250 250 250 F F F 16 320-339 144 144 144 251 251 251 T F T <--- 17 340-359 141 141 141 247 247 247 T F T <--- 18 360-379 147 147 147 254 254 254 F F F 19 380-399 147 147 147 254 254 254 F F F 20 400-419 147 147 147 254 254 254 F F F 21 420-439 147 147 147 254 254 254 F F F 22 440-459 142 142 142 248 248 248 T F T <--- 23 460-479 146 146 146 253 253 253 F F F 24 480-499 146 146 146 253 253 253 T F T <--- 25 500-519 146 146 146 253 253 253 F F F 26 520-539 140 140 140 246 246 246 T F T <--- 27 540-559 141 141 141 247 247 247 F F F 28 560-579 142 142 142 248 248 248 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 147, Max flush: 254 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 74 Verifying data now No change after verify Loop: 74 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 148 255 255 255 F F F 1 020-039 148 148 148 255 255 255 F F F 2 040-059 148 148 148 255 255 255 F F F 3 060-079 148 148 148 255 255 255 F F F 4 080-099 148 148 148 255 255 255 F F F 5 100-119 146 146 146 253 253 253 F F F 6 120-139 134 134 134 234 234 234 F F F 7 140-159 148 148 148 255 255 255 F F F 8 160-179 147 147 147 254 254 254 F F F 9 180-199 148 148 148 255 255 255 F F F 10 200-219 142 142 142 248 248 248 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 145 145 145 252 252 252 F F F 13 260-279 148 148 148 255 255 255 F F F 14 280-299 142 142 142 248 248 248 F F F 15 300-319 144 144 144 250 250 250 F F F 16 320-339 148 148 148 255 255 255 F F F 17 340-359 148 148 148 255 255 255 F F F 18 360-379 147 147 147 254 254 254 F F F 19 380-399 147 147 147 254 254 254 F F F 20 400-419 147 147 147 254 254 254 F F F 21 420-439 147 147 147 254 254 254 F F F 22 440-459 148 148 148 255 255 255 F F F 23 460-479 146 146 146 253 253 253 F F F 24 480-499 148 148 148 255 255 255 F F F 25 500-519 146 146 146 253 253 253 F F F 26 520-539 148 148 148 255 255 255 F F F 27 540-559 141 141 141 247 247 247 F F F 28 560-579 142 142 142 248 248 248 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 148, Max flush: 255 Begin loop 75 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.380188628Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.388329833Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.388628897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.388657282Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.38866599Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.388677558Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733} {"msg":"Crucible f26782b2-be24-47f3-b93d-caf8f0b1abab has session id: e1687348-94e8-4f77-a9c6-c687c2790dc5","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389010836Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"Upstairs opts: Upstairs UUID: f26782b2-be24-47f3-b93d-caf8f0b1abab, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389033892Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"Crucible stats registered with UUID: f26782b2-be24-47f3-b93d-caf8f0b1abab","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.38904392Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:38.389052978Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389199521Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"io task","client":"1","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389217567Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"io task","client":"0","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389227515Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"io task","client":"2","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389390835Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"task":"crutest"} {"msg":"The guest has requested activation with gen:150","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389407542Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733} {"msg":"f26782b2-be24-47f3-b93d-caf8f0b1abab active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389442765Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389452273Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"client":"0","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389461981Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"client":"1","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.38947169Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"client":"2","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389481248Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"io task","client":"2","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389491306Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"io task","client":"1","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389501214Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"io task","client":"0","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"ds_connection connected from Ok(127.0.0.1:42702)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389511492Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"io task","client":"2","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"ds_connection connected from Ok(127.0.0.1:37244)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389651516Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"io task","client":"1","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"ds_connection connected from Ok(127.0.0.1:37451)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.389676551Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"io task","client":"0","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390239776Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"client":"1","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390292036Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"client":"0","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390359143Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"client":"2","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390560556Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390570204Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"[0]R flush_numbers[0..12]: [255, 255, 255, 255, 255, 253, 234, 255, 254, 255, 248, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390657767Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"downstairs","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"[0]R generation[0..12]: [148, 148, 148, 148, 148, 146, 134, 148, 147, 148, 142, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390669015Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"downstairs","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390679813Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"downstairs","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"[1]R flush_numbers[0..12]: [255, 255, 255, 255, 255, 253, 234, 255, 254, 255, 248, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390690261Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"downstairs","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"[1]R generation[0..12]: [148, 148, 148, 148, 148, 146, 134, 148, 147, 148, 142, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390700739Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"downstairs","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390711467Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"downstairs","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"[2]R flush_numbers[0..12]: [255, 255, 255, 255, 255, 253, 234, 255, 254, 255, 248, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390721945Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"downstairs","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"[2]R generation[0..12]: [148, 148, 148, 148, 148, 146, 134, 148, 147, 148, 142, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390732224Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"downstairs","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390742942Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"downstairs","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"Max found gen is 149","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.39075242Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"downstairs","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"Generation requested: 150 >= found:149","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390762168Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"downstairs","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"Next flush: 256","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390773776Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"downstairs","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390783094Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"":"downstairs","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390792712Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390801541Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"client":"0","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390811099Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"client":"1","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390820817Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"client":"2","":"downstairs","session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390830555Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390839653Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"f26782b2-be24-47f3-b93d-caf8f0b1abab is now active with session: e1687348-94e8-4f77-a9c6-c687c2790dc5","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390849522Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.3908581Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"session_id":"e1687348-94e8-4f77-a9c6-c687c2790dc5"} {"msg":"The guest has finished waiting for activation with:150","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390867398Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733} {"msg":"Activated sub_volume f26782b2-be24-47f3-b93d-caf8f0b1abab","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390875417Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.390883615Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.391213093Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"task":"crutest"} 01/30 Write block 77 len 16384 data: 7 8 8 7 02/30 Write block 387 len 28672 data: 8 9 7 9 11 12 15 03/30 Read block 153 len 12288 04/30 Write block 363 len 40960 data: 5 8 9 8 6 5 6 7 11 11 05/30 Read block 115 len 28672 06/30 Read block 296 len 24576 07/30 Flush 08/30 Write block 149 len 20480 data: 7 7 9 10 8 09/30 Write block 209 len 20480 data: 15 18 15 16 15 10/30 Read block 490 len 28672 11/30 Write block 102 len 8192 data: 8 10 12/30 Read block 298 len 40960 13/30 Read block 196 len 16384 14/30 Read block 520 len 32768 15/30 Flush 16/30 Write block 155 len 32768 data: 8 9 8 8 9 6 8 7 17/30 Read block 435 len 4096 18/30 Flush 19/30 Write block 316 len 12288 data: 14 15 13 20/30 Write block 257 len 12288 data: 14 18 15 21/30 Read block 142 len 28672 22/30 Flush 23/30 Write block 502 len 20480 data: 12 14 15 13 10 24/30 Write block 492 len 36864 data: 10 10 10 9 7 9 11 12 12 25/30 Write block 536 len 4096 data: 7 26/30 Read block 81 len 8192 27/30 Read block 365 len 20480 28/30 Write block 49 len 36864 data: 9 7 7 6 6 9 13 16 14 29/30 Write block 137 len 12288 data: 16 19 21 30/30 Write block 153 len 8192 data: 9 7 ---------------------------------------------------------------- Crucible gen:150 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Write 10 Done Done Done false 1004 Acked Read 7 Done Done Sent false 1005 Acked Read 6 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 5 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Read 7 Done Done Sent false 1010 Acked Write 2 Done Done Sent false 1011 Acked Read 10 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Read 8 Done Done Sent false 1014 Acked Flush 0 Done Done Sent false 1015 Acked Write 8 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Flush 0 Done Done Sent false 1018 Acked Write 3 Done Done Sent false 1019 Acked Write 3 Done Done Sent false 1020 Acked Read 7 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 5 Done Done Sent false 1023 Acked Write 9 Done Done Sent false 1024 Acked Write 1 Done Done Sent false 1025 Acked Read 2 Done Done Sent false 1026 Acked Read 5 Done Done Sent false 1027 Acked Write 9 Sent Sent Sent false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 26 32 Done 27 27 4 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1021)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:38.399275713Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:38.399498982Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1733,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 148 255 255 255 F F F 1 020-039 148 148 148 255 255 255 F F F 2 040-059 148 148 148 255 255 255 T T F <--- 3 060-079 150 150 148 256 256 255 F F T <--- 4 080-099 150 150 148 256 256 255 F F T <--- 5 100-119 150 150 146 257 257 253 F F F <--- 6 120-139 134 134 134 234 234 234 T T F <--- 7 140-159 150 150 148 258 258 255 T T F <--- 8 160-179 150 150 147 258 258 254 F F F <--- 9 180-199 148 148 148 255 255 255 F F F 10 200-219 150 150 142 257 257 248 F F F <--- 11 220-239 145 145 145 252 252 252 F F F 12 240-259 150 150 145 259 259 252 F F F <--- 13 260-279 148 148 148 255 255 255 F F F 14 280-299 142 142 142 248 248 248 F F F 15 300-319 150 150 144 259 259 250 F F F <--- 16 320-339 148 148 148 255 255 255 F F F 17 340-359 148 148 148 255 255 255 F F F 18 360-379 150 150 147 256 256 254 F F T <--- 19 380-399 150 150 147 256 256 254 F F T <--- 20 400-419 147 147 147 254 254 254 F F F 21 420-439 147 147 147 254 254 254 F F F 22 440-459 148 148 148 255 255 255 F F F 23 460-479 146 146 146 253 253 253 F F F 24 480-499 148 148 148 255 255 255 T T F <--- 25 500-519 146 146 146 253 253 253 T T F <--- 26 520-539 148 148 148 255 255 255 T T F <--- 27 540-559 141 141 141 247 247 247 F F F 28 560-579 142 142 142 248 248 248 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 150, Max flush: 259 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 75 Verifying data now No change after verify Loop: 75 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 148 255 255 255 F F F 1 020-039 148 148 148 255 255 255 F F F 2 040-059 151 151 151 260 260 260 F F F 3 060-079 150 150 150 256 256 256 F F F 4 080-099 150 150 150 256 256 256 F F F 5 100-119 150 150 150 257 257 257 F F F 6 120-139 151 151 151 260 260 260 F F F 7 140-159 151 151 151 260 260 260 F F F 8 160-179 150 150 150 258 258 258 F F F 9 180-199 148 148 148 255 255 255 F F F 10 200-219 150 150 150 257 257 257 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 150 150 150 259 259 259 F F F 13 260-279 148 148 148 255 255 255 F F F 14 280-299 142 142 142 248 248 248 F F F 15 300-319 150 150 150 259 259 259 F F F 16 320-339 148 148 148 255 255 255 F F F 17 340-359 148 148 148 255 255 255 F F F 18 360-379 150 150 150 256 256 256 F F F 19 380-399 150 150 150 256 256 256 F F F 20 400-419 147 147 147 254 254 254 F F F 21 420-439 147 147 147 254 254 254 F F F 22 440-459 148 148 148 255 255 255 F F F 23 460-479 146 146 146 253 253 253 F F F 24 480-499 151 151 151 260 260 260 F F F 25 500-519 151 151 151 260 260 260 F F F 26 520-539 151 151 151 260 260 260 F F F 27 540-559 141 141 141 247 247 247 F F F 28 560-579 142 142 142 248 248 248 F F F 29 580-599 147 147 147 254 254 254 F F F Max gen: 151, Max flush: 260 Begin loop 76 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.65896457Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.667000075Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.667330813Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.667389802Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.667410218Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.667422506Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741} {"msg":"Crucible 53f6d924-038b-4cc5-bc37-54afb745d5a8 has session id: c390f599-28f4-4736-ac04-fb9ac4c323eb","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.667733288Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"Upstairs opts: Upstairs UUID: 53f6d924-038b-4cc5-bc37-54afb745d5a8, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.667763122Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"Crucible stats registered with UUID: 53f6d924-038b-4cc5-bc37-54afb745d5a8","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.6677738Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:42.667783379Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.667920323Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"io task","client":"1","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.66793831Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"io task","client":"0","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.667948678Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"io task","client":"2","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.668113027Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"task":"crutest"} {"msg":"The guest has requested activation with gen:152","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.668131963Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741} {"msg":"53f6d924-038b-4cc5-bc37-54afb745d5a8 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.668160998Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.668170386Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"client":"0","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.668181874Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"client":"1","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.668191492Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"client":"2","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.66820095Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"io task","client":"2","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.668210659Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"io task","client":"0","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.668220467Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"io task","client":"1","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:46187)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.668260569Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"io task","client":"2","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:37177)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.668271467Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"io task","client":"1","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"ds_connection connected from Ok(127.0.0.1:61219)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.668305821Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"io task","client":"0","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.668956909Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"client":"1","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.668993252Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"client":"0","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669011819Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"client":"2","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669245405Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.66927057Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"[0]R flush_numbers[0..12]: [255, 255, 260, 256, 256, 257, 260, 260, 258, 255, 257, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669318691Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"downstairs","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"[0]R generation[0..12]: [148, 148, 151, 150, 150, 150, 151, 151, 150, 148, 150, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669330329Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"downstairs","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669343426Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"downstairs","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"[1]R flush_numbers[0..12]: [255, 255, 260, 256, 256, 257, 260, 260, 258, 255, 257, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669353994Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"downstairs","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"[1]R generation[0..12]: [148, 148, 151, 150, 150, 150, 151, 151, 150, 148, 150, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669364542Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"downstairs","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.66937532Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"downstairs","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"[2]R flush_numbers[0..12]: [255, 255, 260, 256, 256, 257, 260, 260, 258, 255, 257, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669385878Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"downstairs","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"[2]R generation[0..12]: [148, 148, 151, 150, 150, 150, 151, 151, 150, 148, 150, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669396247Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"downstairs","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669406925Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"downstairs","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"Max found gen is 152","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669416473Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"downstairs","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"Generation requested: 152 >= found:152","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669426081Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"downstairs","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"Next flush: 261","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669435329Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"downstairs","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669444697Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"":"downstairs","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669454456Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669463404Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"client":"0","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669473102Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"client":"1","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.66948393Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"client":"2","":"downstairs","session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669493688Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669502667Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"53f6d924-038b-4cc5-bc37-54afb745d5a8 is now active with session: c390f599-28f4-4736-ac04-fb9ac4c323eb","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669512575Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669521213Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"session_id":"c390f599-28f4-4736-ac04-fb9ac4c323eb"} {"msg":"The guest has finished waiting for activation with:152","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669530341Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741} {"msg":"Activated sub_volume 53f6d924-038b-4cc5-bc37-54afb745d5a8","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.66953838Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669546398Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.669862299Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"task":"crutest"} 01/30 Write block 591 len 28672 data: 13 15 17 16 14 12 8 02/30 Write block 576 len 20480 data: 9 11 13 13 13 03/30 Write block 347 len 8192 data: 7 7 04/30 Read block 329 len 16384 05/30 Write block 525 len 20480 data: 14 12 12 11 10 06/30 Write block 200 len 24576 data: 12 10 11 11 9 9 07/30 Read block 305 len 16384 08/30 Write block 59 len 40960 data: 15 15 14 15 9 11 13 15 15 14 09/30 Read block 431 len 24576 10/30 Read block 47 len 24576 11/30 Read block 555 len 28672 12/30 Read block 455 len 24576 13/30 Flush 14/30 Write block 113 len 12288 data: 15 14 12 15/30 Read block 162 len 16384 16/30 Flush 17/30 Write block 597 len 12288 data: 9 7 3 18/30 Write block 354 len 40960 data: 11 10 11 13 9 11 10 5 5 6 19/30 Write block 255 len 36864 data: 13 14 15 19 16 19 19 16 14 20/30 Read block 114 len 20480 21/30 Write block 456 len 8192 data: 13 12 22/30 Read block 32 len 32768 23/30 Read block 354 len 24576 24/30 Read block 395 len 32768 25/30 Write block 414 len 8192 data: 12 12 26/30 Write block 473 len 24576 data: 7 7 8 8 7 8 27/30 Write block 24 len 8192 data: 13 13 28/30 Read block 421 len 4096 29/30 Write block 549 len 32768 data: 10 10 11 13 12 12 11 11 30/30 Read block 400 len 8192 ---------------------------------------------------------------- Crucible gen:152 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 5 Done Done Done false 1002 Acked Write 2 Done Done Done false 1003 Acked Read 4 Done Done Done false 1004 Acked Write 5 Done Done Done false 1005 Acked Write 6 Done Done Done false 1006 Acked Read 4 Done Done Sent false 1007 Acked Write 10 Done Done Sent false 1008 Acked Read 6 Done Done Sent false 1009 Acked Read 6 Done Done Sent false 1010 Acked Read 7 Done Done Sent false 1011 Acked Read 6 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 3 Done Done Sent false 1014 Acked Read 4 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 3 Done Done Sent false 1017 Acked Write 10 Done Done Sent false 1018 Acked Write 9 Done Done Sent false 1019 Acked Read 5 Done Done Sent false 1020 Acked Write 2 Done Done Sent false 1021 Acked Read 8 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Write 2 Done Done Sent false 1025 Acked Write 6 Done Done Sent false 1026 Acked Write 2 Done Done Sent false 1027 Acked Read 1 Done Done Sent false 1028 Acked Write 8 Done Done Sent false 1029 Acked Read 2 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 24 24 Done 30 30 6 66 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1015)) Some(JobId(1015)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:42.678671789Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:42.678854195Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1741,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 148 255 255 255 F F F 1 020-039 148 148 148 255 255 255 T T F <--- 2 040-059 152 152 151 261 261 260 F F F <--- 3 060-079 152 152 150 261 261 256 F F F <--- 4 080-099 150 150 150 256 256 256 F F F 5 100-119 152 152 150 262 262 257 F F F <--- 6 120-139 151 151 151 260 260 260 F F F 7 140-159 151 151 151 260 260 260 F F F 8 160-179 150 150 150 258 258 258 F F F 9 180-199 148 148 148 255 255 255 F F F 10 200-219 152 152 150 261 261 257 F F T <--- 11 220-239 145 145 145 252 252 252 F F F 12 240-259 150 150 150 259 259 259 T T F <--- 13 260-279 148 148 148 255 255 255 T T F <--- 14 280-299 142 142 142 248 248 248 F F F 15 300-319 150 150 150 259 259 259 F F F 16 320-339 148 148 148 255 255 255 F F F 17 340-359 152 152 148 261 261 255 T T T <--- 18 360-379 150 150 150 256 256 256 T T F <--- 19 380-399 150 150 150 256 256 256 F F F 20 400-419 147 147 147 254 254 254 T T F <--- 21 420-439 147 147 147 254 254 254 F F F 22 440-459 148 148 148 255 255 255 T T F <--- 23 460-479 146 146 146 253 253 253 T T F <--- 24 480-499 151 151 151 260 260 260 F F F 25 500-519 151 151 151 260 260 260 F F F 26 520-539 152 152 151 261 261 260 F F T <--- 27 540-559 141 141 141 247 247 247 T T F <--- 28 560-579 152 152 142 261 261 248 F F T <--- 29 580-599 152 152 147 261 261 254 T T T <--- Max gen: 152, Max flush: 262 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 76 Verifying data now No change after verify Loop: 76 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 148 148 148 255 255 255 F F F 1 020-039 153 153 153 263 263 263 F F F 2 040-059 152 152 152 261 261 261 F F F 3 060-079 152 152 152 261 261 261 F F F 4 080-099 150 150 150 256 256 256 F F F 5 100-119 152 152 152 262 262 262 F F F 6 120-139 151 151 151 260 260 260 F F F 7 140-159 151 151 151 260 260 260 F F F 8 160-179 150 150 150 258 258 258 F F F 9 180-199 148 148 148 255 255 255 F F F 10 200-219 152 152 152 261 261 261 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 153 153 153 263 263 263 F F F 13 260-279 153 153 153 263 263 263 F F F 14 280-299 142 142 142 248 248 248 F F F 15 300-319 150 150 150 259 259 259 F F F 16 320-339 148 148 148 255 255 255 F F F 17 340-359 153 153 153 263 263 263 F F F 18 360-379 153 153 153 263 263 263 F F F 19 380-399 150 150 150 256 256 256 F F F 20 400-419 153 153 153 263 263 263 F F F 21 420-439 147 147 147 254 254 254 F F F 22 440-459 153 153 153 263 263 263 F F F 23 460-479 153 153 153 263 263 263 F F F 24 480-499 151 151 151 260 260 260 F F F 25 500-519 151 151 151 260 260 260 F F F 26 520-539 152 152 152 261 261 261 F F F 27 540-559 153 153 153 263 263 263 F F F 28 560-579 152 152 152 261 261 261 F F F 29 580-599 153 153 153 263 263 263 F F F Max gen: 153, Max flush: 263 Begin loop 77 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.941068059Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.949113152Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.949400598Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.949427503Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.949436142Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.94944745Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749} {"msg":"Crucible 13fba12c-06c0-46fa-9f43-4044b36476bf has session id: 0460492a-0fba-457e-896c-c16db76cec8c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.949809862Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"Upstairs opts: Upstairs UUID: 13fba12c-06c0-46fa-9f43-4044b36476bf, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.949838336Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"Crucible stats registered with UUID: 13fba12c-06c0-46fa-9f43-4044b36476bf","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.949849094Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:46.949858223Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950000026Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"io task","client":"0","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950016753Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"io task","client":"2","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950026761Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"io task","client":"1","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950176123Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"task":"crutest"} {"msg":"The guest has requested activation with gen:154","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950188161Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749} {"msg":"13fba12c-06c0-46fa-9f43-4044b36476bf active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950203978Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950212826Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"client":"0","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950222475Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"client":"1","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950232053Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"client":"2","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950241511Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"io task","client":"2","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950251489Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"io task","client":"0","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950263417Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"io task","client":"1","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"ds_connection connected from Ok(127.0.0.1:51384)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950339113Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"io task","client":"2","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"ds_connection connected from Ok(127.0.0.1:63716)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950412859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"io task","client":"1","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"ds_connection connected from Ok(127.0.0.1:55629)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.950425806Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"io task","client":"0","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.95110054Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"client":"0","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951125575Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"client":"1","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951169227Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"client":"2","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951417011Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951457453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"[0]R flush_numbers[0..12]: [255, 263, 261, 261, 256, 262, 260, 260, 258, 255, 261, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951509453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"downstairs","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"[0]R generation[0..12]: [148, 153, 152, 152, 150, 152, 151, 151, 150, 148, 152, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951520681Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"downstairs","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951531619Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"downstairs","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"[1]R flush_numbers[0..12]: [255, 263, 261, 261, 256, 262, 260, 260, 258, 255, 261, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951542177Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"downstairs","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"[1]R generation[0..12]: [148, 153, 152, 152, 150, 152, 151, 151, 150, 148, 152, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951552675Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"downstairs","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951563433Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"downstairs","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"[2]R flush_numbers[0..12]: [255, 263, 261, 261, 256, 262, 260, 260, 258, 255, 261, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951576381Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"downstairs","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"[2]R generation[0..12]: [148, 153, 152, 152, 150, 152, 151, 151, 150, 148, 152, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951586869Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"downstairs","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951597597Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"downstairs","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"Max found gen is 154","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951607015Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"downstairs","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"Generation requested: 154 >= found:154","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951616643Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"downstairs","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"Next flush: 264","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951625942Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"downstairs","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.95163524Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"":"downstairs","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951644868Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951653746Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"client":"0","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951663645Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"client":"1","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951673633Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"client":"2","":"downstairs","session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951683721Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951692869Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"13fba12c-06c0-46fa-9f43-4044b36476bf is now active with session: 0460492a-0fba-457e-896c-c16db76cec8c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951702697Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951711436Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"session_id":"0460492a-0fba-457e-896c-c16db76cec8c"} {"msg":"The guest has finished waiting for activation with:154","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951721924Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749} {"msg":"Activated sub_volume 13fba12c-06c0-46fa-9f43-4044b36476bf","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951729892Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.951737761Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.952076197Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"task":"crutest"} 01/30 Write block 139 len 12288 data: 22 19 18 02/30 Write block 252 len 28672 data: 15 13 12 14 15 16 20 03/30 Read block 354 len 20480 04/30 Read block 552 len 8192 05/30 Read block 489 len 24576 06/30 Read block 262 len 40960 07/30 Write block 347 len 4096 data: 8 08/30 Write block 383 len 8192 data: 14 11 09/30 Write block 522 len 28672 data: 13 13 15 15 13 13 12 10/30 Read block 327 len 8192 11/30 Write block 285 len 32768 data: 7 7 8 10 13 13 13 14 12/30 Write block 9 len 4096 data: 12 13/30 Read block 63 len 20480 14/30 Flush 15/30 Flush 16/30 Write block 426 len 24576 data: 7 8 7 8 9 6 17/30 Read block 579 len 36864 18/30 Read block 426 len 8192 19/30 Read block 128 len 12288 20/30 Write block 292 len 4096 data: 15 21/30 Flush 22/30 Write block 245 len 4096 data: 15 23/30 Read block 449 len 20480 24/30 Read block 279 len 40960 25/30 Read block 170 len 12288 26/30 Write block 479 len 12288 data: 9 6 5 27/30 Read block 206 len 32768 28/30 Read block 586 len 28672 29/30 Read block 271 len 24576 30/30 Write block 426 len 16384 data: 8 9 8 9 ---------------------------------------------------------------- Crucible gen:154 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Sent Done false 1001 Acked Write 7 Done Sent Done false 1002 Acked Read 5 Done Sent Done false 1003 Acked Read 2 Done Sent Done false 1004 Acked Read 6 Done Sent Done false 1005 Acked Read 10 Done Sent Done false 1006 Acked Write 1 Done Sent Done false 1007 Acked Write 2 Done Sent Done false 1008 Acked Write 7 Done Sent Done false 1009 Acked Read 2 Done Sent Done false 1010 Acked Write 8 Done Sent Done false 1011 Acked Write 1 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 6 Done Sent Done false 1016 Acked Read 9 Done Sent Done false 1017 Acked Read 2 Done Sent Done false 1018 Acked Read 3 Done Sent Done false 1019 Acked Write 1 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Write 1 Done Sent Done false 1022 Acked Read 5 Done Sent Done false 1023 Acked Read 10 Done Sent Done false 1024 Acked Read 3 Done Sent Done false 1025 Acked Write 3 Done Sent Done false 1026 Acked Read 8 Done Sent Done false 1027 Acked Read 7 Done Sent Done false 1028 Acked Read 6 Done Sent Done false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 30 1 32 Done 29 0 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) None Some(JobId(1020)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:46.960448749Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:46.960646692Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1749,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 154 148 154 264 255 264 F F F <--- 1 020-039 153 153 153 263 263 263 F F F 2 040-059 152 152 152 261 261 261 F F F 3 060-079 152 152 152 261 261 261 F F F 4 080-099 150 150 150 256 256 256 F F F 5 100-119 152 152 152 262 262 262 F F F 6 120-139 154 151 154 264 260 264 F F F <--- 7 140-159 154 151 154 264 260 264 F F F <--- 8 160-179 150 150 150 258 258 258 F F F 9 180-199 148 148 148 255 255 255 F F F 10 200-219 152 152 152 261 261 261 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 154 153 154 264 263 264 T F T <--- 13 260-279 153 153 153 263 263 263 F F F 14 280-299 154 142 154 266 248 266 F F F <--- 15 300-319 150 150 150 259 259 259 F F F 16 320-339 148 148 148 255 255 255 F F F 17 340-359 154 153 154 264 263 264 F F F <--- 18 360-379 153 153 153 263 263 263 F F F 19 380-399 154 150 154 264 256 264 F F F <--- 20 400-419 153 153 153 263 263 263 F F F 21 420-439 154 147 154 266 254 266 T F T <--- 22 440-459 153 153 153 263 263 263 F F F 23 460-479 153 153 153 263 263 263 T F T <--- 24 480-499 151 151 151 260 260 260 T F T <--- 25 500-519 151 151 151 260 260 260 F F F 26 520-539 154 152 154 264 261 264 F F F <--- 27 540-559 153 153 153 263 263 263 F F F 28 560-579 152 152 152 261 261 261 F F F 29 580-599 153 153 153 263 263 263 F F F Max gen: 154, Max flush: 266 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 77 Verifying data now No change after verify Loop: 77 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 154 154 154 264 264 264 F F F 1 020-039 153 153 153 263 263 263 F F F 2 040-059 152 152 152 261 261 261 F F F 3 060-079 152 152 152 261 261 261 F F F 4 080-099 150 150 150 256 256 256 F F F 5 100-119 152 152 152 262 262 262 F F F 6 120-139 154 154 154 264 264 264 F F F 7 140-159 154 154 154 264 264 264 F F F 8 160-179 150 150 150 258 258 258 F F F 9 180-199 148 148 148 255 255 255 F F F 10 200-219 152 152 152 261 261 261 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 155 155 155 267 267 267 F F F 13 260-279 153 153 153 263 263 263 F F F 14 280-299 154 154 154 266 266 266 F F F 15 300-319 150 150 150 259 259 259 F F F 16 320-339 148 148 148 255 255 255 F F F 17 340-359 154 154 154 264 264 264 F F F 18 360-379 153 153 153 263 263 263 F F F 19 380-399 154 154 154 264 264 264 F F F 20 400-419 153 153 153 263 263 263 F F F 21 420-439 155 155 155 267 267 267 F F F 22 440-459 153 153 153 263 263 263 F F F 23 460-479 155 155 155 267 267 267 F F F 24 480-499 155 155 155 267 267 267 F F F 25 500-519 151 151 151 260 260 260 F F F 26 520-539 154 154 154 264 264 264 F F F 27 540-559 153 153 153 263 263 263 F F F 28 560-579 152 152 152 261 261 261 F F F 29 580-599 153 153 153 263 263 263 F F F Max gen: 155, Max flush: 267 Begin loop 78 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.207011665Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.215069466Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.21536551Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.215400684Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.215410182Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.21542177Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757} {"msg":"Crucible b6b06993-d607-4dc8-b1f1-97b722e9b739 has session id: 7d3abb11-a13f-4914-8c01-1f0857258336","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.215726602Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"Upstairs opts: Upstairs UUID: b6b06993-d607-4dc8-b1f1-97b722e9b739, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.215790351Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"Crucible stats registered with UUID: b6b06993-d607-4dc8-b1f1-97b722e9b739","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.215801298Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:51.215810357Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.21595136Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"io task","client":"1","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.215981205Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"io task","client":"0","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.215992433Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"io task","client":"2","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216129437Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"task":"crutest"} {"msg":"The guest has requested activation with gen:156","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216153782Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757} {"msg":"b6b06993-d607-4dc8-b1f1-97b722e9b739 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216171289Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216180268Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"client":"0","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216189956Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"client":"1","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216199574Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"client":"2","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216209122Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"io task","client":"2","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.21621909Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"io task","client":"0","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216228838Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"io task","client":"1","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"ds_connection connected from Ok(127.0.0.1:59184)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216313503Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"io task","client":"1","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"ds_connection connected from Ok(127.0.0.1:47830)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216338708Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"io task","client":"0","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"ds_connection connected from Ok(127.0.0.1:65227)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216349976Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"io task","client":"2","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216935956Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"client":"2","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216961551Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"client":"0","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.216982237Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"client":"1","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217249307Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217273013Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"[0]R flush_numbers[0..12]: [264, 263, 261, 261, 256, 262, 264, 264, 258, 255, 261, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217326053Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"downstairs","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"[0]R generation[0..12]: [154, 153, 152, 152, 150, 152, 154, 154, 150, 148, 152, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217337701Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"downstairs","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217348779Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"downstairs","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"[1]R flush_numbers[0..12]: [264, 263, 261, 261, 256, 262, 264, 264, 258, 255, 261, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217359407Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"downstairs","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"[1]R generation[0..12]: [154, 153, 152, 152, 150, 152, 154, 154, 150, 148, 152, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217369735Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"downstairs","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217380593Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"downstairs","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"[2]R flush_numbers[0..12]: [264, 263, 261, 261, 256, 262, 264, 264, 258, 255, 261, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217391231Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"downstairs","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"[2]R generation[0..12]: [154, 153, 152, 152, 150, 152, 154, 154, 150, 148, 152, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217401709Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"downstairs","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217412557Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"downstairs","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"Max found gen is 156","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217422035Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"downstairs","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"Generation requested: 156 >= found:156","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217434713Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"downstairs","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"Next flush: 268","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217443991Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"downstairs","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217453349Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"":"downstairs","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217463377Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217472056Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"client":"0","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217482024Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"client":"1","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217491782Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"client":"2","":"downstairs","session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.21750189Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217511198Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"b6b06993-d607-4dc8-b1f1-97b722e9b739 is now active with session: 7d3abb11-a13f-4914-8c01-1f0857258336","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217521106Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217529735Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"session_id":"7d3abb11-a13f-4914-8c01-1f0857258336"} {"msg":"The guest has finished waiting for activation with:156","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217538783Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757} {"msg":"Activated sub_volume b6b06993-d607-4dc8-b1f1-97b722e9b739","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217546732Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.2175547Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.217906174Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"task":"crutest"} 01/30 Write block 498 len 32768 data: 12 13 13 10 13 15 16 14 02/30 Read block 267 len 4096 03/30 Write block 591 len 32768 data: 14 16 18 17 15 13 10 8 04/30 Read block 405 len 16384 05/30 Flush 06/30 Write block 25 len 8192 data: 14 13 07/30 Write block 549 len 40960 data: 11 11 12 14 13 13 12 12 14 12 08/30 Write block 469 len 4096 data: 8 09/30 Write block 418 len 40960 data: 12 13 11 12 12 9 9 7 9 10 10/30 Read block 593 len 24576 11/30 Read block 206 len 16384 12/30 Read block 230 len 36864 13/30 Read block 119 len 20480 14/30 Write block 55 len 4096 data: 14 15/30 Write block 191 len 32768 data: 14 16 20 18 18 18 15 13 16/30 Write block 308 len 20480 data: 15 14 17 16 14 17/30 Flush 18/30 Write block 58 len 32768 data: 13 16 16 15 16 10 12 14 19/30 Read block 246 len 24576 20/30 Flush 21/30 Write block 92 len 32768 data: 8 10 10 8 8 6 4 6 22/30 Write block 506 len 4096 data: 11 23/30 Write block 513 len 24576 data: 9 12 14 15 15 14 24/30 Write block 437 len 40960 data: 11 8 9 11 12 12 12 10 11 9 25/30 Read block 42 len 12288 26/30 Read block 389 len 24576 27/30 Read block 552 len 16384 28/30 Read block 537 len 24576 29/30 Read block 325 len 28672 30/30 Read block 102 len 28672 ---------------------------------------------------------------- Crucible gen:156 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Done Done Done false 1001 Acked Read 1 Done Sent Done false 1002 Acked Write 8 Done Sent Done false 1003 Acked Read 4 Done Sent Done false 1004 Acked Flush 0 Done Sent Done false 1005 Acked Write 2 Done Sent Done false 1006 Acked Write 10 Done Sent Done false 1007 Acked Write 1 Done Sent Done false 1008 Acked Write 10 Done Sent Done false 1009 Acked Read 6 Done Sent Done false 1010 Acked Read 4 Done Sent Done false 1011 Acked Read 9 Done Sent Done false 1012 Acked Read 5 Done Sent Done false 1013 Acked Write 1 Done Sent Done false 1014 Acked Write 8 Done Sent Done false 1015 Acked Write 5 Done Sent Done false 1016 Acked Flush 0 Done Sent Done false 1017 Acked Write 8 Done Sent Done false 1018 Acked Read 6 Done Sent Done false 1019 Acked Flush 0 Done Sent Done false 1020 Acked Write 8 Done Sent Done false 1021 Acked Write 1 Done Sent Done false 1022 Acked Write 6 Done Sent Done false 1023 Acked Write 10 Done Sent Done false 1024 Acked Read 3 Done Sent Done false 1025 Acked Read 6 Done Sent Done false 1026 Acked Read 4 Done Sent Done false 1027 Acked Read 6 Done Sent Done false 1028 Acked Read 7 Done Sent Done false 1029 Acked Read 7 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 29 1 30 Done 30 1 29 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) None Some(JobId(1019)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:51.226551175Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:51.226752288Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1757,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 154 154 154 264 264 264 F F F 1 020-039 156 153 156 269 263 269 F F F <--- 2 040-059 156 152 156 270 261 270 F F F <--- 3 060-079 156 152 156 270 261 270 F F F <--- 4 080-099 150 150 150 256 256 256 T F T <--- 5 100-119 152 152 152 262 262 262 F F F 6 120-139 154 154 154 264 264 264 F F F 7 140-159 154 154 154 264 264 264 F F F 8 160-179 150 150 150 258 258 258 F F F 9 180-199 156 148 156 269 255 269 F F F <--- 10 200-219 152 152 152 261 261 261 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 155 155 155 267 267 267 F F F 13 260-279 153 153 153 263 263 263 F F F 14 280-299 154 154 154 266 266 266 F F F 15 300-319 156 150 156 269 259 269 F F F <--- 16 320-339 148 148 148 255 255 255 F F F 17 340-359 154 154 154 264 264 264 F F F 18 360-379 153 153 153 263 263 263 F F F 19 380-399 154 154 154 264 264 264 F F F 20 400-419 156 153 156 269 263 269 F F F <--- 21 420-439 156 155 156 269 267 269 T F T <--- 22 440-459 153 153 153 263 263 263 T F T <--- 23 460-479 156 155 156 269 267 269 F F F <--- 24 480-499 156 155 156 268 267 268 F T F <--- 25 500-519 156 151 156 268 260 268 T T T <--- 26 520-539 154 154 154 264 264 264 F F F 27 540-559 156 153 156 269 263 269 F F F <--- 28 560-579 152 152 152 261 261 261 F F F 29 580-599 156 153 156 268 263 268 F F F <--- Max gen: 156, Max flush: 270 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 78 Verifying data now No change after verify Loop: 78 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 154 154 154 264 264 264 F F F 1 020-039 156 156 156 269 269 269 F F F 2 040-059 156 156 156 270 270 270 F F F 3 060-079 156 156 156 270 270 270 F F F 4 080-099 157 157 157 271 271 271 F F F 5 100-119 152 152 152 262 262 262 F F F 6 120-139 154 154 154 264 264 264 F F F 7 140-159 154 154 154 264 264 264 F F F 8 160-179 150 150 150 258 258 258 F F F 9 180-199 156 156 156 269 269 269 F F F 10 200-219 152 152 152 261 261 261 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 155 155 155 267 267 267 F F F 13 260-279 153 153 153 263 263 263 F F F 14 280-299 154 154 154 266 266 266 F F F 15 300-319 156 156 156 269 269 269 F F F 16 320-339 148 148 148 255 255 255 F F F 17 340-359 154 154 154 264 264 264 F F F 18 360-379 153 153 153 263 263 263 F F F 19 380-399 154 154 154 264 264 264 F F F 20 400-419 156 156 156 269 269 269 F F F 21 420-439 157 157 157 271 271 271 F F F 22 440-459 157 157 157 271 271 271 F F F 23 460-479 156 156 156 269 269 269 F F F 24 480-499 156 156 156 268 268 268 F F F 25 500-519 157 157 157 271 271 271 F F F 26 520-539 154 154 154 264 264 264 F F F 27 540-559 156 156 156 269 269 269 F F F 28 560-579 152 152 152 261 261 261 F F F 29 580-599 156 156 156 268 268 268 F F F Max gen: 157, Max flush: 271 Begin loop 79 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.478393722Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.486543036Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.486848088Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.486888761Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.486898019Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.486909587Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765} {"msg":"Crucible 87318a4a-c7f3-4c4f-a758-0c0b64db6b1c has session id: 16b1eed3-1b65-45f7-befd-0ab8094df3d1","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487245824Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"Upstairs opts: Upstairs UUID: 87318a4a-c7f3-4c4f-a758-0c0b64db6b1c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487275078Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"Crucible stats registered with UUID: 87318a4a-c7f3-4c4f-a758-0c0b64db6b1c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487285986Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:55.487295405Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487427Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"io task","client":"2","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487444167Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"io task","client":"0","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487454365Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"io task","client":"1","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487607426Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"task":"crutest"} {"msg":"The guest has requested activation with gen:158","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487623043Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765} {"msg":"87318a4a-c7f3-4c4f-a758-0c0b64db6b1c active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.4876399Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487650888Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"client":"0","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487660646Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"client":"1","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487670384Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"client":"2","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487679823Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"io task","client":"2","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487689781Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"io task","client":"0","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487699749Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"io task","client":"1","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"ds_connection connected from Ok(127.0.0.1:62045)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487738992Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"io task","client":"2","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"ds_connection connected from Ok(127.0.0.1:39332)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.48774994Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"io task","client":"0","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"ds_connection connected from Ok(127.0.0.1:45378)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.487764867Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"io task","client":"1","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.48844118Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"client":"0","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488453488Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"client":"1","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488464376Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"client":"2","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.48875926Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488768789Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"[0]R flush_numbers[0..12]: [264, 269, 270, 270, 271, 262, 264, 264, 258, 269, 261, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.48881274Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"downstairs","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"[0]R generation[0..12]: [154, 156, 156, 156, 157, 152, 154, 154, 150, 156, 152, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488825488Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"downstairs","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488836356Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"downstairs","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"[1]R flush_numbers[0..12]: [264, 269, 270, 270, 271, 262, 264, 264, 258, 269, 261, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488846884Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"downstairs","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"[1]R generation[0..12]: [154, 156, 156, 156, 157, 152, 154, 154, 150, 156, 152, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488857282Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"downstairs","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.48886813Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"downstairs","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"[2]R flush_numbers[0..12]: [264, 269, 270, 270, 271, 262, 264, 264, 258, 269, 261, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488878658Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"downstairs","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"[2]R generation[0..12]: [154, 156, 156, 156, 157, 152, 154, 154, 150, 156, 152, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488889146Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"downstairs","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488900024Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"downstairs","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"Max found gen is 158","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488909522Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"downstairs","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"Generation requested: 158 >= found:158","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488919151Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"downstairs","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"Next flush: 272","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488928599Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"downstairs","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488937967Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"":"downstairs","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488947855Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488956604Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"client":"0","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488967432Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"client":"1","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.48897731Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"client":"2","":"downstairs","session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488987078Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.488996016Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"87318a4a-c7f3-4c4f-a758-0c0b64db6b1c is now active with session: 16b1eed3-1b65-45f7-befd-0ab8094df3d1","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.489005914Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.489014583Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"session_id":"16b1eed3-1b65-45f7-befd-0ab8094df3d1"} {"msg":"The guest has finished waiting for activation with:158","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.489023611Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765} {"msg":"Activated sub_volume 87318a4a-c7f3-4c4f-a758-0c0b64db6b1c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.489031619Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.489039878Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.489377585Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"task":"crutest"} 01/30 Write block 22 len 16384 data: 12 13 14 15 02/30 Read block 495 len 4096 03/30 Read block 211 len 8192 04/30 Write block 485 len 4096 data: 10 05/30 Write block 51 len 8192 data: 8 7 06/30 Write block 554 len 4096 data: 14 07/30 Write block 321 len 8192 data: 13 9 08/30 Read block 51 len 36864 09/30 Write block 56 len 12288 data: 17 15 14 10/30 Read block 162 len 8192 11/30 Write block 174 len 16384 data: 14 12 11 12 12/30 Write block 301 len 20480 data: 12 15 17 20 16 13/30 Flush 14/30 Write block 573 len 4096 data: 8 15/30 Write block 153 len 12288 data: 10 8 9 16/30 Flush 17/30 Write block 260 len 40960 data: 20 20 17 15 9 7 7 10 8 10 18/30 Write block 565 len 16384 data: 12 11 11 9 19/30 Write block 173 len 32768 data: 10 15 13 12 13 16 13 11 20/30 Read block 455 len 12288 21/30 Read block 467 len 8192 22/30 Write block 23 len 36864 data: 14 15 16 14 11 14 16 22 23 23/30 Read block 456 len 8192 24/30 Write block 386 len 32768 data: 9 9 10 8 10 12 13 16 25/30 Flush 26/30 Write block 574 len 16384 data: 7 9 10 12 27/30 Read block 398 len 40960 28/30 Read block 488 len 36864 29/30 Read block 349 len 28672 30/30 Read block 350 len 8192 ---------------------------------------------------------------- Crucible gen:158 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 1 Done Done Sent false 1002 Acked Read 2 Done Done Sent false 1003 Acked Write 1 Done Done Sent false 1004 Acked Write 2 Done Done Sent false 1005 Acked Write 1 Done Done Sent false 1006 Acked Write 2 Done Done Sent false 1007 Acked Read 9 Done Done Sent false 1008 Acked Write 3 Done Done Sent false 1009 Acked Read 2 Done Done Sent false 1010 Acked Write 4 Done Done Sent false 1011 Acked Write 5 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 1 Done Done Sent false 1014 Acked Write 3 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 10 Done Done Sent false 1017 Acked Write 4 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Read 3 Done Done Sent false 1020 Acked Read 2 Done Done Sent false 1021 Acked Write 9 Done Done Sent false 1022 Acked Read 2 Done Done Sent false 1023 Acked Write 8 Done Done Sent false 1024 Acked Flush 0 Done Done Sent false 1025 Acked Write 4 Done Done Sent false 1026 Acked Read 10 Done Done Sent false 1027 Acked Read 9 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Read 2 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 30 29 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1024)) Some(JobId(1024)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:55.497130843Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:55.497320847Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1765,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 154 154 154 264 264 264 F F F 1 020-039 158 158 156 274 274 269 F F T <--- 2 040-059 158 158 156 272 272 270 F F F <--- 3 060-079 156 156 156 270 270 270 F F F 4 080-099 157 157 157 271 271 271 F F F 5 100-119 152 152 152 262 262 262 F F F 6 120-139 154 154 154 264 264 264 F F F 7 140-159 158 158 154 273 273 264 F F F <--- 8 160-179 158 158 150 274 274 258 F F F <--- 9 180-199 158 158 156 274 274 269 F F F <--- 10 200-219 152 152 152 261 261 261 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 155 155 155 267 267 267 F F F 13 260-279 158 158 153 274 274 263 F F F <--- 14 280-299 154 154 154 266 266 266 F F F 15 300-319 158 158 156 272 272 269 F F F <--- 16 320-339 158 158 148 272 272 255 F F F <--- 17 340-359 154 154 154 264 264 264 F F F 18 360-379 153 153 153 263 263 263 F F F 19 380-399 158 158 154 274 274 264 F F F <--- 20 400-419 156 156 156 269 269 269 F F F 21 420-439 157 157 157 271 271 271 F F F 22 440-459 157 157 157 271 271 271 F F F 23 460-479 156 156 156 269 269 269 F F F 24 480-499 158 158 156 272 272 268 F F F <--- 25 500-519 157 157 157 271 271 271 F F F 26 520-539 154 154 154 264 264 264 F F F 27 540-559 158 158 156 272 272 269 F F F <--- 28 560-579 158 158 152 274 274 261 T T F <--- 29 580-599 156 156 156 268 268 268 F F F Max gen: 158, Max flush: 274 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 79 Verifying data now No change after verify Loop: 79 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 154 154 154 264 264 264 F F F 1 020-039 158 158 158 274 274 274 F F F 2 040-059 158 158 158 272 272 272 F F F 3 060-079 156 156 156 270 270 270 F F F 4 080-099 157 157 157 271 271 271 F F F 5 100-119 152 152 152 262 262 262 F F F 6 120-139 154 154 154 264 264 264 F F F 7 140-159 158 158 158 273 273 273 F F F 8 160-179 158 158 158 274 274 274 F F F 9 180-199 158 158 158 274 274 274 F F F 10 200-219 152 152 152 261 261 261 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 155 155 155 267 267 267 F F F 13 260-279 158 158 158 274 274 274 F F F 14 280-299 154 154 154 266 266 266 F F F 15 300-319 158 158 158 272 272 272 F F F 16 320-339 158 158 158 272 272 272 F F F 17 340-359 154 154 154 264 264 264 F F F 18 360-379 153 153 153 263 263 263 F F F 19 380-399 158 158 158 274 274 274 F F F 20 400-419 156 156 156 269 269 269 F F F 21 420-439 157 157 157 271 271 271 F F F 22 440-459 157 157 157 271 271 271 F F F 23 460-479 156 156 156 269 269 269 F F F 24 480-499 158 158 158 272 272 272 F F F 25 500-519 157 157 157 271 271 271 F F F 26 520-539 154 154 154 264 264 264 F F F 27 540-559 158 158 158 272 272 272 F F F 28 560-579 159 159 159 275 275 275 F F F 29 580-599 156 156 156 268 268 268 F F F Max gen: 159, Max flush: 275 Begin loop 80 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.741717199Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.749851625Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.75014809Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750175924Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750184643Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.75019801Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773} {"msg":"Crucible 55377a65-020d-43be-a58e-eae3d5a0e3cb has session id: cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750513341Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"Upstairs opts: Upstairs UUID: 55377a65-020d-43be-a58e-eae3d5a0e3cb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750539246Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"Crucible stats registered with UUID: 55377a65-020d-43be-a58e-eae3d5a0e3cb","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750549284Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:59.750558243Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750708535Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"io task","client":"1","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750739379Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"io task","client":"0","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750750297Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"io task","client":"2","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750889111Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"task":"crutest"} {"msg":"The guest has requested activation with gen:160","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750913356Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773} {"msg":"55377a65-020d-43be-a58e-eae3d5a0e3cb active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750938472Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.75094792Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"client":"0","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750957718Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"client":"1","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750967216Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"client":"2","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750976504Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"io task","client":"2","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.750986563Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"io task","client":"0","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.751011998Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"io task","client":"1","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"ds_connection connected from Ok(127.0.0.1:56010)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.751025805Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"io task","client":"2","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"ds_connection connected from Ok(127.0.0.1:47175)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.751039773Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"io task","client":"1","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"ds_connection connected from Ok(127.0.0.1:60070)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.751050021Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"io task","client":"0","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.751723974Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"client":"0","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.75174528Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"client":"2","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.751807559Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"client":"1","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.75206567Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752088656Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"[0]R flush_numbers[0..12]: [264, 274, 272, 270, 271, 262, 264, 273, 274, 274, 261, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752156254Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"downstairs","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"[0]R generation[0..12]: [154, 158, 158, 156, 157, 152, 154, 158, 158, 158, 152, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752167911Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"downstairs","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752179079Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"downstairs","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"[1]R flush_numbers[0..12]: [264, 274, 272, 270, 271, 262, 264, 273, 274, 274, 261, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752189847Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"downstairs","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"[1]R generation[0..12]: [154, 158, 158, 156, 157, 152, 154, 158, 158, 158, 152, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752200305Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"downstairs","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752213453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"downstairs","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"[2]R flush_numbers[0..12]: [264, 274, 272, 270, 271, 262, 264, 273, 274, 274, 261, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752223991Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"downstairs","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"[2]R generation[0..12]: [154, 158, 158, 156, 157, 152, 154, 158, 158, 158, 152, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752234369Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"downstairs","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752245027Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"downstairs","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"Max found gen is 160","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752254305Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"downstairs","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"Generation requested: 160 >= found:160","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752263953Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"downstairs","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"Next flush: 276","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752273272Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"downstairs","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.75228266Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"":"downstairs","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752292378Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752301026Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"client":"0","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752310625Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"client":"1","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752320083Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"client":"2","":"downstairs","session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752329831Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752338859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"55377a65-020d-43be-a58e-eae3d5a0e3cb is now active with session: cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752348647Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752359016Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"session_id":"cb9ee9e6-dc7a-45cc-b1a2-88cf75fa23fa"} {"msg":"The guest has finished waiting for activation with:160","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752367994Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773} {"msg":"Activated sub_volume 55377a65-020d-43be-a58e-eae3d5a0e3cb","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752375822Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752383801Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.752726587Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"task":"crutest"} 01/30 Write block 437 len 24576 data: 12 9 10 12 13 13 02/30 Read block 245 len 32768 03/30 Read block 591 len 8192 04/30 Read block 125 len 16384 05/30 Write block 401 len 32768 data: 11 12 12 15 12 12 9 8 06/30 Read block 301 len 8192 07/30 Write block 572 len 28672 data: 7 9 8 10 11 13 14 08/30 Read block 265 len 16384 09/30 Write block 170 len 20480 data: 8 8 10 11 16 10/30 Read block 591 len 36864 11/30 Read block 52 len 4096 12/30 Write block 475 len 24576 data: 9 9 8 9 10 7 13/30 Flush 14/30 Write block 383 len 16384 data: 15 12 8 10 15/30 Write block 64 len 8192 data: 13 15 16/30 Read block 486 len 24576 17/30 Read block 18 len 40960 18/30 Read block 2 len 36864 19/30 Read block 443 len 36864 20/30 Write block 292 len 36864 data: 16 13 12 10 10 10 12 11 13 21/30 Read block 455 len 40960 22/30 Read block 443 len 36864 23/30 Flush 24/30 Flush 25/30 Write block 209 len 12288 data: 16 19 16 26/30 Read block 129 len 32768 27/30 Write block 590 len 40960 data: 15 15 17 19 18 16 14 11 9 4 28/30 Read block 215 len 4096 29/30 Write block 8 len 12288 data: 12 13 15 30/30 Write block 289 len 24576 data: 14 14 14 17 14 13 ---------------------------------------------------------------- Crucible gen:160 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 6 Done Done Done false 1001 Acked Read 8 Done Done Done false 1002 Acked Read 2 Done Sent Done false 1003 Acked Read 4 Done Sent Done false 1004 Acked Write 8 Done Sent Done false 1005 Acked Read 2 Done Sent Done false 1006 Acked Write 7 Done Sent Done false 1007 Acked Read 4 Done Sent Done false 1008 Acked Write 5 Done Sent Done false 1009 Acked Read 9 Done Sent Done false 1010 Acked Read 1 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Flush 0 Done Sent Done false 1013 Acked Write 4 Done Sent Done false 1014 Acked Write 2 Done Sent Done false 1015 Acked Read 6 Done Sent Done false 1016 Acked Read 10 Done Sent Done false 1017 Acked Read 9 Done Sent Done false 1018 Acked Read 9 Done Sent Done false 1019 Acked Write 9 Done Sent Done false 1020 Acked Read 10 Done Sent Done false 1021 Acked Read 9 Done Sent Done false 1022 Acked Flush 0 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 3 Done Sent Done false 1025 Acked Read 8 Done Sent Done false 1026 Acked Write 10 Done Sent Done false 1027 Acked Read 1 Done Sent Done false 1028 Acked Write 3 Sent Sent Sent false 1029 Acked Write 6 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 28 2 32 Done 28 2 28 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) None Some(JobId(1023)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:01:59.762257672Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:01:59.762438988Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1773,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 154 154 154 264 264 264 T F T <--- 1 020-039 158 158 158 274 274 274 F F F 2 040-059 158 158 158 272 272 272 F F F 3 060-079 160 156 160 277 270 277 F F F <--- 4 080-099 157 157 157 271 271 271 F F F 5 100-119 152 152 152 262 262 262 F F F 6 120-139 154 154 154 264 264 264 F F F 7 140-159 158 158 158 273 273 273 F F F 8 160-179 160 158 160 276 274 276 F F F <--- 9 180-199 158 158 158 274 274 274 F F F 10 200-219 152 152 152 261 261 261 T F T <--- 11 220-239 145 145 145 252 252 252 F F F 12 240-259 155 155 155 267 267 267 F F F 13 260-279 158 158 158 274 274 274 F F F 14 280-299 160 154 160 277 266 277 T F T <--- 15 300-319 160 158 160 277 272 277 F F F <--- 16 320-339 158 158 158 272 272 272 F F F 17 340-359 154 154 154 264 264 264 F F F 18 360-379 153 153 153 263 263 263 F F F 19 380-399 160 158 160 277 274 277 F F F <--- 20 400-419 160 156 160 276 269 276 F F F <--- 21 420-439 160 157 160 276 271 276 F T F <--- 22 440-459 160 157 160 276 271 276 F T F <--- 23 460-479 160 156 160 276 269 276 F F F <--- 24 480-499 160 158 160 276 272 276 F F F <--- 25 500-519 157 157 157 271 271 271 F F F 26 520-539 154 154 154 264 264 264 F F F 27 540-559 158 158 158 272 272 272 F F F 28 560-579 160 159 160 276 275 276 F F F <--- 29 580-599 156 156 156 268 268 268 T F T <--- Max gen: 160, Max flush: 277 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 80 Verifying data now No change after verify Loop: 80 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 161 161 161 278 278 278 F F F 1 020-039 158 158 158 274 274 274 F F F 2 040-059 158 158 158 272 272 272 F F F 3 060-079 160 160 160 277 277 277 F F F 4 080-099 157 157 157 271 271 271 F F F 5 100-119 152 152 152 262 262 262 F F F 6 120-139 154 154 154 264 264 264 F F F 7 140-159 158 158 158 273 273 273 F F F 8 160-179 160 160 160 276 276 276 F F F 9 180-199 158 158 158 274 274 274 F F F 10 200-219 161 161 161 278 278 278 F F F 11 220-239 145 145 145 252 252 252 F F F 12 240-259 155 155 155 267 267 267 F F F 13 260-279 158 158 158 274 274 274 F F F 14 280-299 161 161 161 278 278 278 F F F 15 300-319 160 160 160 277 277 277 F F F 16 320-339 158 158 158 272 272 272 F F F 17 340-359 154 154 154 264 264 264 F F F 18 360-379 153 153 153 263 263 263 F F F 19 380-399 160 160 160 277 277 277 F F F 20 400-419 160 160 160 276 276 276 F F F 21 420-439 160 160 160 276 276 276 F F F 22 440-459 160 160 160 276 276 276 F F F 23 460-479 160 160 160 276 276 276 F F F 24 480-499 160 160 160 276 276 276 F F F 25 500-519 157 157 157 271 271 271 F F F 26 520-539 154 154 154 264 264 264 F F F 27 540-559 158 158 158 272 272 272 F F F 28 560-579 160 160 160 276 276 276 F F F 29 580-599 161 161 161 278 278 278 F F F Max gen: 161, Max flush: 278 Begin loop 81 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.012516734Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.020528104Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.020831717Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.020859162Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.020871859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.020883207Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781} {"msg":"Crucible 989e60b6-3767-403f-8a6a-8a5542aeeb55 has session id: 6796aebd-578d-45bb-8cd9-fa105ec5c7b2","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021250369Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"Upstairs opts: Upstairs UUID: 989e60b6-3767-403f-8a6a-8a5542aeeb55, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021278703Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"Crucible stats registered with UUID: 989e60b6-3767-403f-8a6a-8a5542aeeb55","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021289671Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:04.021301059Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021433044Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"io task","client":"1","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021449411Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"io task","client":"0","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021459419Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"io task","client":"2","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021602853Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"task":"crutest"} {"msg":"The guest has requested activation with gen:162","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021627668Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781} {"msg":"989e60b6-3767-403f-8a6a-8a5542aeeb55 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021651583Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021660902Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"client":"0","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.0216705Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"client":"1","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021680328Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"client":"2","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021689846Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"io task","client":"2","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021699774Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"io task","client":"0","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021716011Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"io task","client":"1","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"ds_connection connected from Ok(127.0.0.1:55926)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021754104Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"io task","client":"1","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"ds_connection connected from Ok(127.0.0.1:51945)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021788178Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"io task","client":"0","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"ds_connection connected from Ok(127.0.0.1:59390)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.021870252Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"io task","client":"2","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022348023Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"client":"1","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022376278Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"client":"0","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022467191Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"client":"2","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.02262985Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022647137Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"[0]R flush_numbers[0..12]: [278, 274, 272, 277, 271, 262, 264, 273, 276, 274, 278, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022727932Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"downstairs","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"[0]R generation[0..12]: [161, 158, 158, 160, 157, 152, 154, 158, 160, 158, 161, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.02273914Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"downstairs","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022750148Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"downstairs","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"[1]R flush_numbers[0..12]: [278, 274, 272, 277, 271, 262, 264, 273, 276, 274, 278, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022760626Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"downstairs","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"[1]R generation[0..12]: [161, 158, 158, 160, 157, 152, 154, 158, 160, 158, 161, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022771224Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"downstairs","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022782112Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"downstairs","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"[2]R flush_numbers[0..12]: [278, 274, 272, 277, 271, 262, 264, 273, 276, 274, 278, 252]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.02279267Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"downstairs","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"[2]R generation[0..12]: [161, 158, 158, 160, 157, 152, 154, 158, 160, 158, 161, 145]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022803118Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"downstairs","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022813966Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"downstairs","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"Max found gen is 162","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022825124Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"downstairs","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"Generation requested: 162 >= found:162","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022834832Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"downstairs","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"Next flush: 279","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022844Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"downstairs","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022853558Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"":"downstairs","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022863206Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022871965Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"client":"0","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022881633Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"client":"1","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022891151Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"client":"2","":"downstairs","session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022900989Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022910008Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"989e60b6-3767-403f-8a6a-8a5542aeeb55 is now active with session: 6796aebd-578d-45bb-8cd9-fa105ec5c7b2","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022919876Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022928534Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"session_id":"6796aebd-578d-45bb-8cd9-fa105ec5c7b2"} {"msg":"The guest has finished waiting for activation with:162","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022937643Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781} {"msg":"Activated sub_volume 989e60b6-3767-403f-8a6a-8a5542aeeb55","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.022945461Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.0229535Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.023296395Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"task":"crutest"} 01/30 Write block 371 len 28672 data: 12 12 10 12 14 12 12 02/30 Read block 195 len 40960 03/30 Flush 04/30 Write block 539 len 16384 data: 6 7 8 6 05/30 Flush 06/30 Write block 217 len 36864 data: 15 14 11 12 11 8 7 8 7 07/30 Read block 313 len 20480 08/30 Flush 09/30 Write block 266 len 36864 data: 8 11 9 11 9 7 6 6 10 10/30 Read block 68 len 20480 11/30 Read block 172 len 24576 12/30 Flush 13/30 Write block 525 len 28672 data: 16 14 14 13 11 7 7 14/30 Read block 75 len 24576 15/30 Read block 529 len 40960 16/30 Write block 187 len 40960 data: 11 10 12 12 15 17 21 19 19 19 17/30 Write block 222 len 16384 data: 9 8 9 8 18/30 Write block 483 len 20480 data: 10 11 11 11 10 19/30 Read block 78 len 16384 20/30 Write block 39 len 4096 data: 14 21/30 Flush 22/30 Write block 428 len 28672 data: 9 10 10 7 6 8 8 23/30 Write block 179 len 12288 data: 14 12 10 24/30 Write block 53 len 16384 data: 7 10 15 18 25/30 Read block 183 len 8192 26/30 Flush 27/30 Write block 197 len 40960 data: 16 14 11 13 11 12 12 10 10 9 28/30 Read block 300 len 12288 29/30 Write block 281 len 4096 data: 9 30/30 Read block 255 len 40960 ---------------------------------------------------------------- Crucible gen:162 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Read 10 Done Done Sent false 1002 Acked Flush 0 Done Done Sent false 1003 Acked Write 4 Done Done Sent false 1004 Acked Flush 0 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Read 5 Done Done Sent false 1007 Acked Flush 0 Done Done Sent false 1008 Acked Write 9 Done Done Sent false 1009 Acked Read 5 Done Done Sent false 1010 Acked Read 6 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 7 Done Done Sent false 1013 Acked Read 6 Done Done Sent false 1014 Acked Read 10 Done Done Sent false 1015 Acked Write 10 Done Done Sent false 1016 Acked Write 4 Done Done Sent false 1017 Acked Write 5 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Write 1 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 7 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Write 4 Done Done Sent false 1024 Acked Read 2 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Read 3 Done Done Sent false 1028 Acked Write 1 Done Done Sent false 1029 Acked Read 10 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 29 30 Done 30 29 1 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:04.032694135Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:04.0328827Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1781,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 161 161 161 278 278 278 F F F 1 020-039 162 162 158 283 283 274 F F F <--- 2 040-059 162 162 158 284 284 272 F F F <--- 3 060-079 160 160 160 277 277 277 F F F 4 080-099 157 157 157 271 271 271 F F F 5 100-119 152 152 152 262 262 262 F F F 6 120-139 154 154 154 264 264 264 F F F 7 140-159 158 158 158 273 273 273 F F F 8 160-179 162 162 160 284 284 276 F F F <--- 9 180-199 162 162 158 284 284 274 T T F <--- 10 200-219 162 162 161 281 281 278 T T F <--- 11 220-239 162 162 145 283 283 252 F F F <--- 12 240-259 155 155 155 267 267 267 F F F 13 260-279 162 162 158 282 282 274 F F F <--- 14 280-299 161 161 161 278 278 278 T T F <--- 15 300-319 160 160 160 277 277 277 F F F 16 320-339 158 158 158 272 272 272 F F F 17 340-359 154 154 154 264 264 264 F F F 18 360-379 162 162 153 279 279 263 F F T <--- 19 380-399 160 160 160 277 277 277 F F F 20 400-419 160 160 160 276 276 276 F F F 21 420-439 162 162 160 284 284 276 F F F <--- 22 440-459 160 160 160 276 276 276 F F F 23 460-479 160 160 160 276 276 276 F F F 24 480-499 162 162 160 283 283 276 F F F <--- 25 500-519 157 157 157 271 271 271 F F F 26 520-539 162 162 154 283 283 264 F F F <--- 27 540-559 162 162 158 280 280 272 F F F <--- 28 560-579 160 160 160 276 276 276 F F F 29 580-599 161 161 161 278 278 278 F F F Max gen: 162, Max flush: 284 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 81 Verifying data now No change after verify Loop: 81 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 161 161 161 278 278 278 F F F 1 020-039 162 162 162 283 283 283 F F F 2 040-059 162 162 162 284 284 284 F F F 3 060-079 160 160 160 277 277 277 F F F 4 080-099 157 157 157 271 271 271 F F F 5 100-119 152 152 152 262 262 262 F F F 6 120-139 154 154 154 264 264 264 F F F 7 140-159 158 158 158 273 273 273 F F F 8 160-179 162 162 162 284 284 284 F F F 9 180-199 163 163 163 285 285 285 F F F 10 200-219 163 163 163 285 285 285 F F F 11 220-239 162 162 162 283 283 283 F F F 12 240-259 155 155 155 267 267 267 F F F 13 260-279 162 162 162 282 282 282 F F F 14 280-299 163 163 163 285 285 285 F F F 15 300-319 160 160 160 277 277 277 F F F 16 320-339 158 158 158 272 272 272 F F F 17 340-359 154 154 154 264 264 264 F F F 18 360-379 162 162 162 279 279 279 F F F 19 380-399 160 160 160 277 277 277 F F F 20 400-419 160 160 160 276 276 276 F F F 21 420-439 162 162 162 284 284 284 F F F 22 440-459 160 160 160 276 276 276 F F F 23 460-479 160 160 160 276 276 276 F F F 24 480-499 162 162 162 283 283 283 F F F 25 500-519 157 157 157 271 271 271 F F F 26 520-539 162 162 162 283 283 283 F F F 27 540-559 162 162 162 280 280 280 F F F 28 560-579 160 160 160 276 276 276 F F F 29 580-599 161 161 161 278 278 278 F F F Max gen: 163, Max flush: 285 Begin loop 82 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.278974823Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.287031944Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.287332088Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.287357303Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.287365692Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.28737733Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789} {"msg":"Crucible 5236165f-bbd9-4d42-8313-0d2ca49fc238 has session id: fa11eac7-1d99-4d3b-87f7-22607170a33d","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.287715386Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"Upstairs opts: Upstairs UUID: 5236165f-bbd9-4d42-8313-0d2ca49fc238, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.287738642Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"Crucible stats registered with UUID: 5236165f-bbd9-4d42-8313-0d2ca49fc238","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.28774842Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:08.287757528Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.287898652Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"io task","client":"2","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.287928696Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"io task","client":"0","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.287939754Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"io task","client":"1","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.2880686Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"task":"crutest"} {"msg":"The guest has requested activation with gen:164","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.288086927Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789} {"msg":"5236165f-bbd9-4d42-8313-0d2ca49fc238 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.288116311Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.288125909Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"client":"0","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.288135628Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"client":"1","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.288145296Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"client":"2","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.288154834Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"io task","client":"2","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.288164642Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"io task","client":"0","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.2881745Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"io task","client":"1","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"ds_connection connected from Ok(127.0.0.1:36932)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.28823082Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"io task","client":"1","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"ds_connection connected from Ok(127.0.0.1:36342)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.288255525Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"io task","client":"0","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"ds_connection connected from Ok(127.0.0.1:34726)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.288267393Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"io task","client":"2","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.288939927Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"client":"0","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.288959323Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"client":"2","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.288970561Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"client":"1","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289247209Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289257138Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"[0]R flush_numbers[0..12]: [278, 283, 284, 277, 271, 262, 264, 273, 284, 285, 285, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.28929686Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"downstairs","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"[0]R generation[0..12]: [161, 162, 162, 160, 157, 152, 154, 158, 162, 163, 163, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289308298Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"downstairs","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289319276Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"downstairs","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"[1]R flush_numbers[0..12]: [278, 283, 284, 277, 271, 262, 264, 273, 284, 285, 285, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289329834Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"downstairs","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"[1]R generation[0..12]: [161, 162, 162, 160, 157, 152, 154, 158, 162, 163, 163, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289340272Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"downstairs","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.28935107Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"downstairs","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"[2]R flush_numbers[0..12]: [278, 283, 284, 277, 271, 262, 264, 273, 284, 285, 285, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289361658Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"downstairs","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"[2]R generation[0..12]: [161, 162, 162, 160, 157, 152, 154, 158, 162, 163, 163, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289372316Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"downstairs","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289383124Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"downstairs","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"Max found gen is 164","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289392532Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"downstairs","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"Generation requested: 164 >= found:164","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.28940219Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"downstairs","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"Next flush: 286","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289411399Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"downstairs","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289420677Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"":"downstairs","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289430645Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289440943Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"client":"0","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289450841Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"client":"1","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289460609Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"client":"2","":"downstairs","session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289470608Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289479746Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"5236165f-bbd9-4d42-8313-0d2ca49fc238 is now active with session: fa11eac7-1d99-4d3b-87f7-22607170a33d","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289489454Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"session_id":"fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289498302Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"session_id":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 fa11eac7-1d99-4d3b-87f7-22607170a33d"} {"msg":"The guest has finished waiting for activation with:164","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.28951248Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789} {"msg":"Activated sub_volume 5236165f-bbd9-4d42-8313-0d2ca49fc238","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289520488Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289528507Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"task":"crutest"} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.289771801Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"task":"crutest"} 01/30 Write block 557 len 28672 data: 15 13 13 12 12 13 13 02/30 Write block 184 len 8192 data: 13 13 03/30 Write block 330 len 12288 data: 16 15 17 04/30 Write block 356 len 40960 data: 12 14 10 12 11 6 6 7 9 10 05/30 Write block 513 len 16384 data: 10 13 15 16 06/30 Write block 382 len 36864 data: 14 16 13 9 11 10 11 9 11 07/30 Read block 102 len 16384 08/30 Read block 370 len 24576 09/30 Flush 10/30 Write block 362 len 8192 data: 7 8 11/30 Write block 269 len 16384 data: 12 10 8 7 12/30 Read block 471 len 40960 13/30 Write block 254 len 16384 data: 13 15 16 17 14/30 Flush 15/30 Write block 191 len 40960 data: 16 18 22 20 20 20 17 15 12 14 16/30 Write block 257 len 8192 data: 18 21 17/30 Write block 529 len 8192 data: 12 8 18/30 Read block 226 len 28672 19/30 Write block 245 len 24576 data: 16 15 16 15 16 13 20/30 Read block 212 len 4096 21/30 Read block 467 len 8192 22/30 Read block 368 len 24576 23/30 Read block 541 len 12288 24/30 Read block 207 len 20480 25/30 Write block 313 len 4096 data: 14 26/30 Read block 379 len 12288 27/30 Write block 342 len 8192 data: 12 13 28/30 Read block 378 len 8192 29/30 Write block 441 len 8192 data: 14 14 30/30 Write block 39 len 28672 data: 15 14 16 17 18 17 18 ---------------------------------------------------------------- Crucible gen:164 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Write 3 Done Done Done false 1003 Acked Write 10 Done Done Done false 1004 Acked Write 4 Done Done Done false 1005 Acked Write 9 Done Sent Done false 1006 Acked Read 4 Done Sent Done false 1007 Acked Read 6 Done Sent Done false 1008 Acked Flush 0 Done Sent Done false 1009 Acked Write 2 Done Sent Done false 1010 Acked Write 4 Done Sent Done false 1011 Acked Read 10 Done Sent Done false 1012 Acked Write 4 Done Sent Done false 1013 Acked Flush 0 Done Sent Done false 1014 Acked Write 10 Done Sent Done false 1015 Acked Write 2 Done Sent Done false 1016 Acked Write 2 Done Sent Done false 1017 Acked Read 7 Done Sent Done false 1018 Acked Write 6 Done Sent Done false 1019 Acked Read 1 Done Sent Done false 1020 Acked Read 2 Done Sent Done false 1021 Acked Read 6 Done Sent Done false 1022 Acked Read 3 Done Sent Done false 1023 Acked Read 5 Done Sent Done false 1024 Acked Write 1 Done Sent Done false 1025 Acked Read 3 Done Sent Done false 1026 Acked Write 2 Done Sent Done false 1027 Acked Read 2 Done Sent Done false 1028 Acked Write 2 Done Sent Sent false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 25 2 28 Done 29 5 28 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1013)) None Some(JobId(1013)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:08.297184483Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:08.297369238Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1789,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 161 161 161 278 278 278 F F F 1 020-039 162 162 162 283 283 283 T F T <--- 2 040-059 162 162 162 284 284 284 T F T <--- 3 060-079 160 160 160 277 277 277 F F F 4 080-099 157 157 157 271 271 271 F F F 5 100-119 152 152 152 262 262 262 F F F 6 120-139 154 154 154 264 264 264 F F F 7 140-159 158 158 158 273 273 273 F F F 8 160-179 162 162 162 284 284 284 F F F 9 180-199 164 163 164 286 285 286 T T T <--- 10 200-219 163 163 163 285 285 285 T F T <--- 11 220-239 162 162 162 283 283 283 F F F 12 240-259 164 155 164 287 267 287 T F T <--- 13 260-279 164 162 164 287 282 287 F F F <--- 14 280-299 163 163 163 285 285 285 F F F 15 300-319 160 160 160 277 277 277 T F T <--- 16 320-339 164 158 164 286 272 286 F T F <--- 17 340-359 164 154 164 286 264 286 T T T <--- 18 360-379 164 162 164 287 279 287 F T F <--- 19 380-399 164 160 164 286 277 286 F F F <--- 20 400-419 160 160 160 276 276 276 F F F 21 420-439 162 162 162 284 284 284 F F F 22 440-459 160 160 160 276 276 276 T F T <--- 23 460-479 160 160 160 276 276 276 F F F 24 480-499 162 162 162 283 283 283 F F F 25 500-519 164 157 164 286 271 286 F T F <--- 26 520-539 162 162 162 283 283 283 T F T <--- 27 540-559 164 162 164 286 280 286 F T F <--- 28 560-579 164 160 164 286 276 286 F T F <--- 29 580-599 161 161 161 278 278 278 F F F Max gen: 164, Max flush: 287 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 82 Verifying data now No change after verify Loop: 82 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 161 161 161 278 278 278 F F F 1 020-039 165 165 165 288 288 288 F F F 2 040-059 165 165 165 288 288 288 F F F 3 060-079 160 160 160 277 277 277 F F F 4 080-099 157 157 157 271 271 271 F F F 5 100-119 152 152 152 262 262 262 F F F 6 120-139 154 154 154 264 264 264 F F F 7 140-159 158 158 158 273 273 273 F F F 8 160-179 162 162 162 284 284 284 F F F 9 180-199 165 165 165 288 288 288 F F F 10 200-219 165 165 165 288 288 288 F F F 11 220-239 162 162 162 283 283 283 F F F 12 240-259 165 165 165 288 288 288 F F F 13 260-279 164 164 164 287 287 287 F F F 14 280-299 163 163 163 285 285 285 F F F 15 300-319 165 165 165 288 288 288 F F F 16 320-339 164 164 164 286 286 286 F F F 17 340-359 165 165 165 288 288 288 F F F 18 360-379 164 164 164 287 287 287 F F F 19 380-399 164 164 164 286 286 286 F F F 20 400-419 160 160 160 276 276 276 F F F 21 420-439 162 162 162 284 284 284 F F F 22 440-459 165 165 165 288 288 288 F F F 23 460-479 160 160 160 276 276 276 F F F 24 480-499 162 162 162 283 283 283 F F F 25 500-519 164 164 164 286 286 286 F F F 26 520-539 165 165 165 288 288 288 F F F 27 540-559 164 164 164 286 286 286 F F F 28 560-579 164 164 164 286 286 286 F F F 29 580-599 161 161 161 278 278 278 F F F Max gen: 165, Max flush: 288 Begin loop 83 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.555399976Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.563430192Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.563735525Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.563766309Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.563776977Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.563788515Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797} {"msg":"Crucible ce496fad-e909-4758-be3c-7c0793b85c5d has session id: d4a0751a-76ad-44fc-97d6-1fba69e9ddb5","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564126712Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"Upstairs opts: Upstairs UUID: ce496fad-e909-4758-be3c-7c0793b85c5d, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564150867Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"Crucible stats registered with UUID: ce496fad-e909-4758-be3c-7c0793b85c5d","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564161195Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:12.564170753Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564312947Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"io task","client":"1","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564329334Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"io task","client":"0","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564339392Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"io task","client":"2","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564490454Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"task":"crutest"} {"msg":"The guest has requested activation with gen:166","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564514529Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797} {"msg":"ce496fad-e909-4758-be3c-7c0793b85c5d active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564538235Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564547453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"client":"0","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564557041Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"client":"1","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564566639Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"client":"2","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564577827Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"io task","client":"2","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564587805Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"io task","client":"0","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564597554Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"io task","client":"1","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"ds_connection connected from Ok(127.0.0.1:40850)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564626778Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"io task","client":"2","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"ds_connection connected from Ok(127.0.0.1:41118)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564664261Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"io task","client":"0","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"ds_connection connected from Ok(127.0.0.1:41379)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.564681918Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"io task","client":"1","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565288354Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"client":"1","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565313649Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"client":"0","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565333496Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"client":"2","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565598026Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565637239Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"[0]R flush_numbers[0..12]: [278, 288, 288, 277, 271, 262, 264, 273, 284, 288, 288, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.56568588Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"downstairs","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"[0]R generation[0..12]: [161, 165, 165, 160, 157, 152, 154, 158, 162, 165, 165, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565697108Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"downstairs","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565708056Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"downstairs","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"[1]R flush_numbers[0..12]: [278, 288, 288, 277, 271, 262, 264, 273, 284, 288, 288, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565718544Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"downstairs","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"[1]R generation[0..12]: [161, 165, 165, 160, 157, 152, 154, 158, 162, 165, 165, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565732081Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"downstairs","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565743069Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"downstairs","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"[2]R flush_numbers[0..12]: [278, 288, 288, 277, 271, 262, 264, 273, 284, 288, 288, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565753667Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"downstairs","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"[2]R generation[0..12]: [161, 165, 165, 160, 157, 152, 154, 158, 162, 165, 165, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565764185Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"downstairs","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565774923Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"downstairs","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"Max found gen is 166","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565784271Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"downstairs","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"Generation requested: 166 >= found:166","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.56579379Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"downstairs","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"Next flush: 289","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565803038Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"downstairs","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565812466Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"":"downstairs","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565822174Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565830753Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"client":"0","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565840271Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"client":"1","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565849719Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"client":"2","":"downstairs","session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565859727Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565868785Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"ce496fad-e909-4758-be3c-7c0793b85c5d is now active with session: d4a0751a-76ad-44fc-97d6-1fba69e9ddb5","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565880023Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565888672Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"session_id":"d4a0751a-76ad-44fc-97d6-1fba69e9ddb5"} {"msg":"The guest has finished waiting for activation with:166","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.56589779Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797} {"msg":"Activated sub_volume ce496fad-e909-4758-be3c-7c0793b85c5d","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565905729Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.565913697Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.566253333Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"task":"crutest"} 01/30 Write block 490 len 32768 data: 12 12 11 11 11 10 8 10 02/30 Read block 391 len 4096 03/30 Write block 473 len 28672 data: 8 8 10 10 9 10 11 04/30 Write block 276 len 20480 data: 11 11 8 10 9 05/30 Write block 191 len 32768 data: 17 19 23 21 21 21 18 16 06/30 Flush 07/30 Write block 448 len 4096 data: 8 08/30 Read block 545 len 16384 09/30 Flush 10/30 Write block 463 len 4096 data: 7 11/30 Read block 355 len 40960 12/30 Write block 78 len 40960 data: 9 9 8 8 9 6 10 13 12 8 13/30 Read block 91 len 16384 14/30 Write block 545 len 32768 data: 13 12 11 11 12 12 13 15 15/30 Read block 554 len 24576 16/30 Flush 17/30 Write block 170 len 16384 data: 9 9 11 12 18/30 Read block 506 len 12288 19/30 Read block 271 len 24576 20/30 Flush 21/30 Write block 16 len 24576 data: 17 13 15 15 13 10 22/30 Read block 400 len 24576 23/30 Write block 125 len 40960 data: 13 16 17 17 17 19 16 16 17 16 24/30 Read block 289 len 20480 25/30 Read block 10 len 20480 26/30 Write block 2 len 32768 data: 4 5 6 7 10 12 13 14 27/30 Write block 100 len 12288 data: 8 8 9 28/30 Read block 311 len 40960 29/30 Read block 267 len 36864 30/30 Write block 186 len 16384 data: 14 12 11 13 ---------------------------------------------------------------- Crucible gen:166 GIO:true work queues: Upstairs:0 downstairs:21 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1009 Acked Write 1 Done Done Done false 1010 Acked Read 10 Done Done Sent false 1011 Acked Write 10 Done Done Sent false 1012 Acked Read 4 Done Done Sent false 1013 Acked Write 8 Done Done Sent false 1014 Acked Read 6 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Write 4 Done Done Sent false 1017 Acked Read 3 Done Done Sent false 1018 Acked Read 6 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 6 Done Done Sent false 1021 Acked Read 6 Done Done Sent false 1022 Acked Write 10 Done Done Sent false 1023 Acked Read 5 Done Done Sent false 1024 Acked Read 5 Done Done Sent false 1025 Acked Write 8 Done Done Sent false 1026 Acked Write 3 Done Done Sent false 1027 Acked Read 10 Done Done Sent false 1028 Acked Read 9 Done Done Sent false 1029 Acked Write 4 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 1 20 22 Done 20 20 1 41 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) Some(JobId(1008)) Downstairs last five completed: 1008 1007 1006 1005 1004 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:12.575846607Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:12.576026923Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1797,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 161 161 161 278 278 278 T T F <--- 1 020-039 165 165 165 288 288 288 T T F <--- 2 040-059 165 165 165 288 288 288 F F F 3 060-079 166 166 160 291 291 277 F F F <--- 4 080-099 166 166 157 291 291 271 F F F <--- 5 100-119 152 152 152 262 262 262 T T F <--- 6 120-139 154 154 154 264 264 264 T T F <--- 7 140-159 158 158 158 273 273 273 F F F 8 160-179 166 166 162 292 292 284 F F F <--- 9 180-199 166 166 166 289 289 289 T T F <--- 10 200-219 165 165 165 288 288 288 F F F 11 220-239 162 162 162 283 283 283 F F F 12 240-259 165 165 165 288 288 288 F F F 13 260-279 166 166 166 289 289 289 F F F 14 280-299 166 166 166 289 289 289 F F F 15 300-319 165 165 165 288 288 288 F F F 16 320-339 164 164 164 286 286 286 F F F 17 340-359 165 165 165 288 288 288 F F F 18 360-379 164 164 164 287 287 287 F F F 19 380-399 164 164 164 286 286 286 F F F 20 400-419 160 160 160 276 276 276 F F F 21 420-439 162 162 162 284 284 284 F F F 22 440-459 166 166 166 290 290 290 F F F 23 460-479 166 166 166 291 291 289 F F T <--- 24 480-499 166 166 166 289 289 289 F F F 25 500-519 164 164 164 286 286 286 F F F 26 520-539 165 165 165 288 288 288 F F F 27 540-559 166 166 164 291 291 286 F F F <--- 28 560-579 164 164 164 286 286 286 F F F 29 580-599 161 161 161 278 278 278 F F F Max gen: 166, Max flush: 292 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 83 Verifying data now No change after verify Loop: 83 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 167 167 167 293 293 293 F F F 1 020-039 167 167 167 293 293 293 F F F 2 040-059 165 165 165 288 288 288 F F F 3 060-079 166 166 166 291 291 291 F F F 4 080-099 166 166 166 291 291 291 F F F 5 100-119 167 167 167 293 293 293 F F F 6 120-139 167 167 167 293 293 293 F F F 7 140-159 158 158 158 273 273 273 F F F 8 160-179 166 166 166 292 292 292 F F F 9 180-199 167 167 167 293 293 293 F F F 10 200-219 165 165 165 288 288 288 F F F 11 220-239 162 162 162 283 283 283 F F F 12 240-259 165 165 165 288 288 288 F F F 13 260-279 166 166 166 289 289 289 F F F 14 280-299 166 166 166 289 289 289 F F F 15 300-319 165 165 165 288 288 288 F F F 16 320-339 164 164 164 286 286 286 F F F 17 340-359 165 165 165 288 288 288 F F F 18 360-379 164 164 164 287 287 287 F F F 19 380-399 164 164 164 286 286 286 F F F 20 400-419 160 160 160 276 276 276 F F F 21 420-439 162 162 162 284 284 284 F F F 22 440-459 166 166 166 290 290 290 F F F 23 460-479 166 166 166 291 291 291 F F F 24 480-499 166 166 166 289 289 289 F F F 25 500-519 164 164 164 286 286 286 F F F 26 520-539 165 165 165 288 288 288 F F F 27 540-559 166 166 166 291 291 291 F F F 28 560-579 164 164 164 286 286 286 F F F 29 580-599 161 161 161 278 278 278 F F F Max gen: 167, Max flush: 293 Begin loop 84 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.824264054Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.832363757Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.832666731Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.832693256Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.832701684Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.832713192Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805} {"msg":"Crucible 311566a6-bbe4-4df9-868d-f98bebc09e83 has session id: 19647ab5-de7c-4e6e-a06a-24b72d9d19b0","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833076624Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"Upstairs opts: Upstairs UUID: 311566a6-bbe4-4df9-868d-f98bebc09e83, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833106918Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"Crucible stats registered with UUID: 311566a6-bbe4-4df9-868d-f98bebc09e83","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833118046Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:16.833129484Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833282175Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"io task","client":"1","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833299132Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"io task","client":"0","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.83330911Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"io task","client":"2","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833467831Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"task":"crutest"} {"msg":"The guest has requested activation with gen:168","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833484518Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805} {"msg":"311566a6-bbe4-4df9-868d-f98bebc09e83 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833518111Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.83352744Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"client":"0","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833537108Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"client":"1","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833546596Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"client":"2","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833555974Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"io task","client":"2","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833565862Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"io task","client":"0","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.83357571Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"io task","client":"1","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"ds_connection connected from Ok(127.0.0.1:55904)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.83363167Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"io task","client":"2","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"ds_connection connected from Ok(127.0.0.1:41643)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833656195Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"io task","client":"1","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"ds_connection connected from Ok(127.0.0.1:50952)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.833667543Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"io task","client":"0","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834297055Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"client":"1","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834321021Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"client":"0","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834367482Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"client":"2","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834632113Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834641681Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"[0]R flush_numbers[0..12]: [293, 293, 288, 291, 291, 293, 293, 273, 292, 293, 288, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834681973Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"downstairs","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"[0]R generation[0..12]: [167, 167, 165, 166, 166, 167, 167, 158, 166, 167, 165, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834693261Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"downstairs","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834704239Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"downstairs","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"[1]R flush_numbers[0..12]: [293, 293, 288, 291, 291, 293, 293, 273, 292, 293, 288, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834714767Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"downstairs","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"[1]R generation[0..12]: [167, 167, 165, 166, 166, 167, 167, 158, 166, 167, 165, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834725335Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"downstairs","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834736233Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"downstairs","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"[2]R flush_numbers[0..12]: [293, 293, 288, 291, 291, 293, 293, 273, 292, 293, 288, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834746741Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"downstairs","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"[2]R generation[0..12]: [167, 167, 165, 166, 166, 167, 167, 158, 166, 167, 165, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834757179Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"downstairs","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834768077Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"downstairs","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"Max found gen is 168","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834779525Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"downstairs","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"Generation requested: 168 >= found:168","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834789293Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"downstairs","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"Next flush: 294","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834798621Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"downstairs","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.83480801Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"":"downstairs","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834818038Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834826796Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"client":"0","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834836364Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"client":"1","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834845943Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"client":"2","":"downstairs","session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834855851Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834864949Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"311566a6-bbe4-4df9-868d-f98bebc09e83 is now active with session: 19647ab5-de7c-4e6e-a06a-24b72d9d19b0","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834874847Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834883586Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"session_id":"19647ab5-de7c-4e6e-a06a-24b72d9d19b0"} {"msg":"The guest has finished waiting for activation with:168","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834892634Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805} {"msg":"Activated sub_volume 311566a6-bbe4-4df9-868d-f98bebc09e83","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.834900572Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 2026-07-20T23:02:16.834908641Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"task":"crutest"} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.835172651Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"task":"crutest"} 01/30 Write block 552 len 28672 data: 16 14 15 13 13 16 14 02/30 Write block 279 len 12288 data: 11 10 10 03/30 Write block 520 len 4096 data: 14 04/30 Read block 255 len 36864 05/30 Write block 344 len 4096 data: 14 06/30 Write block 240 len 20480 data: 12 12 13 14 13 07/30 Read block 111 len 40960 08/30 Read block 7 len 4096 09/30 Read block 410 len 12288 10/30 Read block 68 len 16384 11/30 Write block 216 len 8192 data: 16 16 12/30 Flush 13/30 Write block 559 len 36864 data: 14 13 13 14 14 10 13 12 12 14/30 Write block 243 len 24576 data: 15 14 17 16 17 16 15/30 Write block 4 len 4096 data: 7 16/30 Read block 260 len 36864 17/30 Write block 422 len 36864 data: 13 10 10 8 10 11 10 11 11 18/30 Write block 489 len 28672 data: 13 13 13 12 12 12 11 19/30 Write block 525 len 12288 data: 17 15 15 20/30 Flush 21/30 Write block 431 len 40960 data: 8 7 9 9 11 9 13 10 11 13 22/30 Read block 375 len 8192 23/30 Read block 92 len 24576 24/30 Write block 114 len 40960 data: 15 13 9 9 12 10 10 12 10 14 25/30 Write block 105 len 24576 data: 10 9 12 11 16 18 26/30 Write block 25 len 4096 data: 17 27/30 Write block 141 len 40960 data: 19 15 12 12 10 8 10 7 8 8 28/30 Write block 314 len 16384 data: 15 13 15 16 29/30 Read block 315 len 28672 30/30 Read block 70 len 16384 ---------------------------------------------------------------- Crucible gen:168 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 3 Done Done Sent false 1002 Acked Write 1 Done Done Sent false 1003 Acked Read 9 Done Done Sent false 1004 Acked Write 1 Done Done Sent false 1005 Acked Write 5 Done Done Sent false 1006 Acked Read 10 Done Done Sent false 1007 Acked Read 1 Done Done Sent false 1008 Acked Read 3 Done Done Sent false 1009 Acked Read 4 Done Done Sent false 1010 Acked Write 2 Done Done Sent false 1011 Acked Flush 0 Done Done Sent false 1012 Acked Write 9 Done Done Sent false 1013 Acked Write 6 Done Done Sent false 1014 Acked Write 1 Done Done Sent false 1015 Acked Read 9 Done Done Sent false 1016 Acked Write 9 Done Done Sent false 1017 Acked Write 7 Done Done Sent false 1018 Acked Write 3 Done Done Sent false 1019 Acked Flush 0 Done Done Sent false 1020 Acked Write 10 Done Done Sent false 1021 Acked Read 2 Done Done Sent false 1022 Acked Read 6 Done Done Sent false 1023 Acked Write 10 Done Done Sent false 1024 Acked Write 6 Done Done Sent false 1025 Acked Write 1 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Write 4 Done Done Sent false 1028 Acked Read 7 Done Done Sent false 1029 Acked Read 4 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 29 29 Done 30 30 1 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1019)) Some(JobId(1019)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:16.843223594Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:16.843391132Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1805,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 168 168 167 295 295 293 F F F <--- 1 020-039 167 167 167 293 293 293 T T F <--- 2 040-059 165 165 165 288 288 288 F F F 3 060-079 166 166 166 291 291 291 F F F 4 080-099 166 166 166 291 291 291 F F F 5 100-119 167 167 167 293 293 293 T T F <--- 6 120-139 167 167 167 293 293 293 T T F <--- 7 140-159 158 158 158 273 273 273 T T F <--- 8 160-179 166 166 166 292 292 292 F F F 9 180-199 167 167 167 293 293 293 F F F 10 200-219 168 168 165 294 294 288 F F F <--- 11 220-239 162 162 162 283 283 283 F F F 12 240-259 168 168 165 295 295 288 F F F <--- 13 260-279 168 168 166 294 294 289 F F F <--- 14 280-299 168 168 166 294 294 289 F F F <--- 15 300-319 165 165 165 288 288 288 T T F <--- 16 320-339 164 164 164 286 286 286 F F F 17 340-359 168 168 165 294 294 288 F F F <--- 18 360-379 164 164 164 287 287 287 F F F 19 380-399 164 164 164 286 286 286 F F F 20 400-419 160 160 160 276 276 276 F F F 21 420-439 168 168 162 295 295 284 T T F <--- 22 440-459 166 166 166 290 290 290 T T F <--- 23 460-479 166 166 166 291 291 291 F F F 24 480-499 168 168 166 295 295 289 F F F <--- 25 500-519 164 164 164 286 286 286 F F F 26 520-539 168 168 165 295 295 288 F F F <--- 27 540-559 168 168 166 295 295 291 F F T <--- 28 560-579 168 168 164 295 295 286 F F F <--- 29 580-599 161 161 161 278 278 278 F F F Max gen: 168, Max flush: 295 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 84 Verifying data now No change after verify Loop: 84 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 168 168 168 295 295 295 F F F 1 020-039 169 169 169 296 296 296 F F F 2 040-059 165 165 165 288 288 288 F F F 3 060-079 166 166 166 291 291 291 F F F 4 080-099 166 166 166 291 291 291 F F F 5 100-119 169 169 169 296 296 296 F F F 6 120-139 169 169 169 296 296 296 F F F 7 140-159 169 169 169 296 296 296 F F F 8 160-179 166 166 166 292 292 292 F F F 9 180-199 167 167 167 293 293 293 F F F 10 200-219 168 168 168 294 294 294 F F F 11 220-239 162 162 162 283 283 283 F F F 12 240-259 168 168 168 295 295 295 F F F 13 260-279 168 168 168 294 294 294 F F F 14 280-299 168 168 168 294 294 294 F F F 15 300-319 169 169 169 296 296 296 F F F 16 320-339 164 164 164 286 286 286 F F F 17 340-359 168 168 168 294 294 294 F F F 18 360-379 164 164 164 287 287 287 F F F 19 380-399 164 164 164 286 286 286 F F F 20 400-419 160 160 160 276 276 276 F F F 21 420-439 169 169 169 296 296 296 F F F 22 440-459 169 169 169 296 296 296 F F F 23 460-479 166 166 166 291 291 291 F F F 24 480-499 168 168 168 295 295 295 F F F 25 500-519 164 164 164 286 286 286 F F F 26 520-539 168 168 168 295 295 295 F F F 27 540-559 168 168 168 295 295 295 F F F 28 560-579 168 168 168 295 295 295 F F F 29 580-599 161 161 161 278 278 278 F F F Max gen: 169, Max flush: 296 Begin loop 85 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.103571767Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.111578537Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.111873112Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.111901997Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.111910505Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.111921983Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813} {"msg":"Crucible 3c6e8df1-ae8e-4fa0-aeb8-5a3a31e44a0a has session id: a0454ce9-07ce-4af3-9e77-a25dad6b1eca","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.11225888Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"Upstairs opts: Upstairs UUID: 3c6e8df1-ae8e-4fa0-aeb8-5a3a31e44a0a, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112286745Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"Crucible stats registered with UUID: 3c6e8df1-ae8e-4fa0-aeb8-5a3a31e44a0a","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112297923Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:21.112307061Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112448595Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"io task","client":"1","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112480659Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"io task","client":"0","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112491566Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"io task","client":"2","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112646058Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"task":"crutest"} {"msg":"The guest has requested activation with gen:170","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112669913Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813} {"msg":"3c6e8df1-ae8e-4fa0-aeb8-5a3a31e44a0a active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112678991Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.11268805Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"client":"0","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112698368Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"client":"1","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112708096Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"client":"2","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112717564Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"io task","client":"2","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112727432Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"io task","client":"0","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.11273716Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"io task","client":"1","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"ds_connection connected from Ok(127.0.0.1:53685)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112799999Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"io task","client":"0","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"ds_connection connected from Ok(127.0.0.1:42009)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112825654Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"io task","client":"1","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"ds_connection connected from Ok(127.0.0.1:56796)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.112837112Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"io task","client":"2","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.11353953Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"client":"1","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.113556437Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"client":"0","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.11364753Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"client":"2","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.113889415Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.113936276Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"[0]R flush_numbers[0..12]: [295, 296, 288, 291, 291, 296, 296, 296, 292, 293, 294, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.113975109Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"downstairs","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"[0]R generation[0..12]: [168, 169, 165, 166, 166, 169, 169, 169, 166, 167, 168, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.113986267Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"downstairs","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.113997165Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"downstairs","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"[1]R flush_numbers[0..12]: [295, 296, 288, 291, 291, 296, 296, 296, 292, 293, 294, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114007783Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"downstairs","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"[1]R generation[0..12]: [168, 169, 165, 166, 166, 169, 169, 169, 166, 167, 168, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114018271Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"downstairs","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114028959Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"downstairs","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"[2]R flush_numbers[0..12]: [295, 296, 288, 291, 291, 296, 296, 296, 292, 293, 294, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114039527Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"downstairs","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"[2]R generation[0..12]: [168, 169, 165, 166, 166, 169, 169, 169, 166, 167, 168, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114049905Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"downstairs","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114060703Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"downstairs","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"Max found gen is 170","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114070001Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"downstairs","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"Generation requested: 170 >= found:170","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114079649Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"downstairs","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"Next flush: 297","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114088947Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"downstairs","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114098176Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"":"downstairs","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114109164Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114117682Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"client":"0","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.11412708Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"client":"1","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114136608Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"client":"2","":"downstairs","session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114146227Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114155365Z","Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"3c6e8df1-ae8e-4fa0-aeb8-5a3a31e44a0a is now active with session: a0454ce9-07ce-4af3-9e77-a25dad6b1eca","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114170772Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.11417963Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"session_id":"a0454ce9-07ce-4af3-9e77-a25dad6b1eca"} {"msg":"The guest has finished waiting for activation with:170","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114188639Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813} {"msg":"Activated sub_volume 3c6e8df1-ae8e-4fa0-aeb8-5a3a31e44a0a","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114196517Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114204466Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"task":"crutest"} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.114432053Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"task":"crutest"} 01/30 Write block 316 len 8192 data: 16 17 02/30 Write block 440 len 40960 data: 14 15 15 13 11 12 10 9 9 10 03/30 Read block 235 len 12288 04/30 Read block 552 len 16384 05/30 Read block 37 len 32768 06/30 Write block 251 len 16384 data: 14 16 14 14 07/30 Flush 08/30 Write block 29 len 36864 data: 17 23 24 22 20 16 14 12 13 09/30 Read block 480 len 28672 10/30 Write block 175 len 20480 data: 14 13 14 17 15 11/30 Write block 320 len 24576 data: 13 14 10 11 12 14 12/30 Read block 353 len 40960 13/30 Write block 437 len 28672 data: 14 11 12 15 16 16 14 14/30 Read block 41 len 12288 15/30 Read block 256 len 36864 16/30 Write block 206 len 24576 data: 10 10 10 17 20 17 17/30 Write block 207 len 8192 data: 11 11 18/30 Write block 35 len 36864 data: 15 13 14 15 16 15 17 18 19 19/30 Read block 5 len 40960 20/30 Write block 451 len 24576 data: 11 11 11 10 9 14 21/30 Flush 22/30 Write block 523 len 8192 data: 14 16 23/30 Write block 558 len 4096 data: 15 24/30 Read block 436 len 32768 25/30 Read block 490 len 16384 26/30 Read block 160 len 12288 27/30 Write block 165 len 28672 data: 10 10 9 6 6 10 10 28/30 Read block 57 len 28672 29/30 Write block 210 len 36864 data: 21 18 17 16 19 19 17 17 15 30/30 Write block 143 len 20480 data: 13 13 11 9 11 ---------------------------------------------------------------- Crucible gen:170 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 10 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Read 4 Done Done Done false 1004 Acked Read 8 Done Done Done false 1005 Acked Write 4 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Read 7 Done Done Sent false 1009 Acked Write 5 Done Done Sent false 1010 Acked Write 6 Done Done Sent false 1011 Acked Read 10 Done Done Sent false 1012 Acked Write 7 Done Done Sent false 1013 Acked Read 3 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Write 6 Done Done Sent false 1016 Acked Write 2 Done Done Sent false 1017 Acked Write 9 Done Done Sent false 1018 Acked Read 10 Done Done Sent false 1019 Acked Write 6 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 2 Done Done Sent false 1022 Acked Write 1 Done Done Sent false 1023 Acked Read 8 Done Done Sent false 1024 Acked Read 4 Done Done Sent false 1025 Acked Read 3 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Read 7 Done Done Sent false 1028 Acked Write 9 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 2 2 25 29 Done 28 28 5 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:21.122591475Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:21.122770221Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1813,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 168 168 168 295 295 295 F F F 1 020-039 170 170 169 298 298 296 F F F <--- 2 040-059 170 170 165 298 298 288 F F F <--- 3 060-079 166 166 166 291 291 291 F F F 4 080-099 166 166 166 291 291 291 F F F 5 100-119 169 169 169 296 296 296 F F F 6 120-139 169 169 169 296 296 296 F F F 7 140-159 169 169 169 296 296 296 T T F <--- 8 160-179 170 170 166 298 298 292 T T F <--- 9 180-199 167 167 167 293 293 293 F F F 10 200-219 170 170 168 298 298 294 T T F <--- 11 220-239 162 162 162 283 283 283 F F F 12 240-259 170 170 168 297 297 295 F F F <--- 13 260-279 168 168 168 294 294 294 F F F 14 280-299 168 168 168 294 294 294 F F F 15 300-319 170 170 169 297 297 296 F F T <--- 16 320-339 170 170 164 298 298 286 F F F <--- 17 340-359 168 168 168 294 294 294 F F F 18 360-379 164 164 164 287 287 287 F F F 19 380-399 164 164 164 286 286 286 F F F 20 400-419 160 160 160 276 276 276 F F F 21 420-439 170 170 169 298 298 296 F F F <--- 22 440-459 170 170 169 298 298 296 F F T <--- 23 460-479 166 166 166 291 291 291 F F F 24 480-499 168 168 168 295 295 295 F F F 25 500-519 164 164 164 286 286 286 F F F 26 520-539 168 168 168 295 295 295 T T F <--- 27 540-559 168 168 168 295 295 295 T T F <--- 28 560-579 168 168 168 295 295 295 F F F 29 580-599 161 161 161 278 278 278 F F F Max gen: 170, Max flush: 298 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 85 Verifying data now No change after verify Loop: 85 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 168 168 168 295 295 295 F F F 1 020-039 170 170 170 298 298 298 F F F 2 040-059 170 170 170 298 298 298 F F F 3 060-079 166 166 166 291 291 291 F F F 4 080-099 166 166 166 291 291 291 F F F 5 100-119 169 169 169 296 296 296 F F F 6 120-139 169 169 169 296 296 296 F F F 7 140-159 171 171 171 299 299 299 F F F 8 160-179 171 171 171 299 299 299 F F F 9 180-199 167 167 167 293 293 293 F F F 10 200-219 171 171 171 299 299 299 F F F 11 220-239 162 162 162 283 283 283 F F F 12 240-259 170 170 170 297 297 297 F F F 13 260-279 168 168 168 294 294 294 F F F 14 280-299 168 168 168 294 294 294 F F F 15 300-319 170 170 170 297 297 297 F F F 16 320-339 170 170 170 298 298 298 F F F 17 340-359 168 168 168 294 294 294 F F F 18 360-379 164 164 164 287 287 287 F F F 19 380-399 164 164 164 286 286 286 F F F 20 400-419 160 160 160 276 276 276 F F F 21 420-439 170 170 170 298 298 298 F F F 22 440-459 170 170 170 298 298 298 F F F 23 460-479 166 166 166 291 291 291 F F F 24 480-499 168 168 168 295 295 295 F F F 25 500-519 164 164 164 286 286 286 F F F 26 520-539 171 171 171 299 299 299 F F F 27 540-559 171 171 171 299 299 299 F F F 28 560-579 168 168 168 295 295 295 F F F 29 580-599 161 161 161 278 278 278 F F F Max gen: 171, Max flush: 299 Begin loop 86 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.370242007Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:25.377338108Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.377393787Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.384258472Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.384523492Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.384548987Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.384557436Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.384568794Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821} {"msg":"Crucible 8a70a306-e78f-47f5-a203-3b470bdf06ce has session id: f869bfe3-7168-4c11-8646-39740ca6b584","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.384936325Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"Upstairs opts: Upstairs UUID: 8a70a306-e78f-47f5-a203-3b470bdf06ce, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.384957491Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"Crucible stats registered with UUID: 8a70a306-e78f-47f5-a203-3b470bdf06ce","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.384967439Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:25.384978007Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.385136027Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"io task","client":"2","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.385166982Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"io task","client":"0","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.38517811Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"io task","client":"1","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.385318143Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"task":"crutest"} {"msg":"The guest has requested activation with gen:172","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.385343009Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821} {"msg":"8a70a306-e78f-47f5-a203-3b470bdf06ce active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.385369564Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.385378862Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"client":"0","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.38539069Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"client":"1","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.385400458Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"client":"2","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.385409916Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"io task","client":"2","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.385419644Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"io task","client":"0","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.385429453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"io task","client":"1","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"ds_connection connected from Ok(127.0.0.1:60463)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.385508428Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"io task","client":"1","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"ds_connection connected from Ok(127.0.0.1:56428)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.385653001Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"io task","client":"0","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"ds_connection connected from Ok(127.0.0.1:43053)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.385666138Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"io task","client":"2","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.38613826Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"client":"1","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386155347Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"client":"2","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.38624407Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"client":"0","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386470728Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.38650945Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"[0]R flush_numbers[0..12]: [295, 298, 298, 291, 291, 296, 296, 299, 299, 293, 299, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386557851Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"downstairs","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"[0]R generation[0..12]: [168, 170, 170, 166, 166, 169, 169, 171, 171, 167, 171, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386568899Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"downstairs","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386581827Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"downstairs","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"[1]R flush_numbers[0..12]: [295, 298, 298, 291, 291, 296, 296, 299, 299, 293, 299, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386592355Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"downstairs","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"[1]R generation[0..12]: [168, 170, 170, 166, 166, 169, 169, 171, 171, 167, 171, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386602793Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"downstairs","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386613501Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"downstairs","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"[2]R flush_numbers[0..12]: [295, 298, 298, 291, 291, 296, 296, 299, 299, 293, 299, 283]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386623939Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"downstairs","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"[2]R generation[0..12]: [168, 170, 170, 166, 166, 169, 169, 171, 171, 167, 171, 162]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386634387Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"downstairs","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386645405Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"downstairs","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"Max found gen is 172","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386654683Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"downstairs","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"Generation requested: 172 >= found:172","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386664451Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"downstairs","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"Next flush: 300","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.38667373Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"downstairs","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386682888Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"":"downstairs","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386692386Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386701284Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"client":"0","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386710883Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"client":"1","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386721581Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"client":"2","":"downstairs","session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386731219Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386740197Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"8a70a306-e78f-47f5-a203-3b470bdf06ce is now active with session: f869bfe3-7168-4c11-8646-39740ca6b584","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386749965Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386758694Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"session_id":"f869bfe3-7168-4c11-8646-39740ca6b584"} {"msg":"The guest has finished waiting for activation with:172","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386767722Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821} {"msg":"Activated sub_volume 8a70a306-e78f-47f5-a203-3b470bdf06ce","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.38677564Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.386783669Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.387123705Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"task":"crutest"} 01/30 Write block 174 len 40960 data: 17 15 14 15 18 16 13 11 14 14 02/30 Write block 6 len 24576 data: 11 13 14 15 16 21 03/30 Write block 69 len 4096 data: 10 04/30 Write block 508 len 20480 data: 6 5 8 10 9 05/30 Read block 434 len 4096 06/30 Write block 230 len 24576 data: 13 10 12 14 14 12 07/30 Read block 430 len 4096 08/30 Read block 62 len 28672 09/30 Read block 248 len 8192 10/30 Read block 173 len 4096 11/30 Write block 260 len 40960 data: 21 21 18 16 10 8 9 12 10 13 12/30 Read block 170 len 12288 13/30 Flush 14/30 Write block 15 len 32768 data: 18 18 14 16 16 14 11 13 15/30 Read block 555 len 24576 16/30 Read block 85 len 4096 17/30 Write block 220 len 36864 data: 13 12 10 9 10 9 7 8 9 18/30 Write block 509 len 28672 data: 6 9 11 10 11 14 16 19/30 Read block 257 len 16384 20/30 Read block 562 len 36864 21/30 Write block 109 len 20480 data: 17 19 16 15 16 22/30 Read block 105 len 36864 23/30 Write block 513 len 36864 data: 12 15 17 17 16 15 14 15 13 24/30 Read block 105 len 24576 25/30 Write block 106 len 36864 data: 10 13 12 18 20 17 16 17 16 26/30 Write block 444 len 4096 data: 12 27/30 Write block 187 len 40960 data: 13 12 14 13 18 20 24 22 22 22 28/30 Write block 189 len 4096 data: 15 29/30 Read block 420 len 40960 30/30 Read block 339 len 36864 ---------------------------------------------------------------- Crucible gen:172 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 10 Done Done Done false 1001 Acked Write 6 Done Done Done false 1002 Acked Write 1 Sent Done Done false 1003 Acked Write 5 Sent Done Done false 1004 Acked Read 1 Sent Done Done false 1005 Acked Write 6 Sent Done Done false 1006 Acked Read 1 Sent Done Done false 1007 Acked Read 7 Sent Done Done false 1008 Acked Read 2 Sent Done Done false 1009 Acked Read 1 Sent Done Done false 1010 Acked Write 10 Sent Done Done false 1011 Acked Read 3 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 8 Sent Done Done false 1014 Acked Read 6 Sent Done Done false 1015 Acked Read 1 Sent Done Done false 1016 Acked Write 9 Sent Done Done false 1017 Acked Write 7 Sent Done Done false 1018 Acked Read 4 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Write 5 Sent Done Done false 1021 Acked Read 9 Sent Done Done false 1022 Acked Write 9 Sent Done Done false 1023 Acked Read 6 Sent Done Done false 1024 Acked Write 9 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Write 10 Sent Done Done false 1027 Acked Write 1 Sent Done Done false 1028 Acked Read 10 Sent Done Done false 1029 Acked Read 9 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 28 0 1 29 Done 2 30 29 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1012)) Some(JobId(1012)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:25.396129849Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:25.396317333Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1821,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 168 172 172 295 300 300 T T T <--- 1 020-039 170 170 170 298 298 298 F T T <--- 2 040-059 170 170 170 298 298 298 F F F 3 060-079 166 172 172 291 300 300 F F F <--- 4 080-099 166 166 166 291 291 291 F F F 5 100-119 169 169 169 296 296 296 F T T <--- 6 120-139 169 169 169 296 296 296 F F F 7 140-159 171 171 171 299 299 299 F F F 8 160-179 171 172 172 299 300 300 T F F <--- 9 180-199 167 172 172 293 300 300 T T T <--- 10 200-219 171 171 171 299 299 299 F F F 11 220-239 162 172 172 283 300 300 F T T <--- 12 240-259 170 170 170 297 297 297 F F F 13 260-279 168 172 172 294 300 300 F F F <--- 14 280-299 168 168 168 294 294 294 F F F 15 300-319 170 170 170 297 297 297 F F F 16 320-339 170 170 170 298 298 298 F F F 17 340-359 168 168 168 294 294 294 F F F 18 360-379 164 164 164 287 287 287 F F F 19 380-399 164 164 164 286 286 286 F F F 20 400-419 160 160 160 276 276 276 F F F 21 420-439 170 170 170 298 298 298 F F F 22 440-459 170 170 170 298 298 298 F T T <--- 23 460-479 166 166 166 291 291 291 F F F 24 480-499 168 168 168 295 295 295 F F F 25 500-519 164 172 172 286 300 300 F T T <--- 26 520-539 171 171 171 299 299 299 F T T <--- 27 540-559 171 171 171 299 299 299 F F F 28 560-579 168 168 168 295 295 295 F F F 29 580-599 161 161 161 278 278 278 F F F Max gen: 172, Max flush: 300 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 86 Verifying data now No change after verify Loop: 86 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 301 301 301 F F F 1 020-039 173 173 173 301 301 301 F F F 2 040-059 170 170 170 298 298 298 F F F 3 060-079 172 172 172 300 300 300 F F F 4 080-099 166 166 166 291 291 291 F F F 5 100-119 173 173 173 301 301 301 F F F 6 120-139 169 169 169 296 296 296 F F F 7 140-159 171 171 171 299 299 299 F F F 8 160-179 172 172 172 300 300 300 F F F 9 180-199 173 173 173 301 301 301 F F F 10 200-219 171 171 171 299 299 299 F F F 11 220-239 173 173 173 301 301 301 F F F 12 240-259 170 170 170 297 297 297 F F F 13 260-279 172 172 172 300 300 300 F F F 14 280-299 168 168 168 294 294 294 F F F 15 300-319 170 170 170 297 297 297 F F F 16 320-339 170 170 170 298 298 298 F F F 17 340-359 168 168 168 294 294 294 F F F 18 360-379 164 164 164 287 287 287 F F F 19 380-399 164 164 164 286 286 286 F F F 20 400-419 160 160 160 276 276 276 F F F 21 420-439 170 170 170 298 298 298 F F F 22 440-459 173 173 173 301 301 301 F F F 23 460-479 166 166 166 291 291 291 F F F 24 480-499 168 168 168 295 295 295 F F F 25 500-519 173 173 173 301 301 301 F F F 26 520-539 173 173 173 301 301 301 F F F 27 540-559 171 171 171 299 299 299 F F F 28 560-579 168 168 168 295 295 295 F F F 29 580-599 161 161 161 278 278 278 F F F Max gen: 173, Max flush: 301 Begin loop 87 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.653102994Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:29.660278021Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.66033623Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.667258353Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.667545739Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.667574254Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.667585972Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.66759747Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829} {"msg":"Crucible 445e8515-a70b-4b25-840f-cee13669e961 has session id: 6ef186ab-bf62-4190-9a21-e313c6304499","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.667894024Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"Upstairs opts: Upstairs UUID: 445e8515-a70b-4b25-840f-cee13669e961, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.667923099Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"Crucible stats registered with UUID: 445e8515-a70b-4b25-840f-cee13669e961","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.667933897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:29.667942885Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.66807546Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"io task","client":"1","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.668107604Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"io task","client":"0","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.668118842Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"io task","client":"2","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.668244529Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"task":"crutest"} {"msg":"The guest has requested activation with gen:174","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.668260166Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829} {"msg":"445e8515-a70b-4b25-840f-cee13669e961 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.668276533Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.668285291Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"client":"0","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.668294989Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"client":"1","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.668304597Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"client":"2","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.668315985Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"io task","client":"2","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.668325853Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"io task","client":"0","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.668335662Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"io task","client":"1","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"ds_connection connected from Ok(127.0.0.1:57771)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.668374844Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"io task","client":"1","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"ds_connection connected from Ok(127.0.0.1:35427)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.668385702Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"io task","client":"0","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"ds_connection connected from Ok(127.0.0.1:44959)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.66839607Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"io task","client":"2","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669050708Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"client":"2","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669076933Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"client":"1","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669108387Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"client":"0","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669321247Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669347312Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"[0]R flush_numbers[0..12]: [301, 301, 298, 300, 291, 301, 296, 299, 300, 301, 299, 301]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669396203Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"downstairs","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"[0]R generation[0..12]: [173, 173, 170, 172, 166, 173, 169, 171, 172, 173, 171, 173]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669407511Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"downstairs","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669418549Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"downstairs","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"[1]R flush_numbers[0..12]: [301, 301, 298, 300, 291, 301, 296, 299, 300, 301, 299, 301]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669429127Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"downstairs","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"[1]R generation[0..12]: [173, 173, 170, 172, 166, 173, 169, 171, 172, 173, 171, 173]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669442384Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"downstairs","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669453082Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"downstairs","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"[2]R flush_numbers[0..12]: [301, 301, 298, 300, 291, 301, 296, 299, 300, 301, 299, 301]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.66946349Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"downstairs","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"[2]R generation[0..12]: [173, 173, 170, 172, 166, 173, 169, 171, 172, 173, 171, 173]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669474028Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"downstairs","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669484816Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"downstairs","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"Max found gen is 174","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669494125Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"downstairs","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"Generation requested: 174 >= found:174","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669503813Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"downstairs","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"Next flush: 302","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669513111Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"downstairs","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669522409Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"":"downstairs","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669531967Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669540556Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"client":"0","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669550084Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"client":"1","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669559652Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"client":"2","":"downstairs","session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.66956949Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669578429Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"445e8515-a70b-4b25-840f-cee13669e961 is now active with session: 6ef186ab-bf62-4190-9a21-e313c6304499","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669589247Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669597865Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"session_id":"6ef186ab-bf62-4190-9a21-e313c6304499"} {"msg":"The guest has finished waiting for activation with:174","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669606953Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829} {"msg":"Activated sub_volume 445e8515-a70b-4b25-840f-cee13669e961","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669614772Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.6696228Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.669894869Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"task":"crutest"} 01/30 Write block 93 len 32768 data: 11 11 9 9 7 5 7 9 02/30 Read block 510 len 28672 03/30 Write block 152 len 20480 data: 11 11 9 10 10 04/30 Read block 10 len 20480 05/30 Read block 380 len 40960 06/30 Read block 243 len 32768 07/30 Read block 495 len 12288 08/30 Read block 137 len 32768 09/30 Read block 394 len 36864 10/30 Write block 571 len 16384 data: 9 8 10 9 11/30 Flush 12/30 Write block 369 len 32768 data: 7 8 13 13 11 13 15 13 13/30 Write block 204 len 36864 data: 11 11 11 12 12 18 22 19 18 14/30 Read block 299 len 20480 15/30 Write block 168 len 32768 data: 7 7 11 11 12 13 18 16 16/30 Read block 223 len 12288 17/30 Write block 594 len 4096 data: 19 18/30 Write block 427 len 12288 data: 12 11 12 19/30 Read block 233 len 8192 20/30 Flush 21/30 Write block 338 len 12288 data: 19 16 14 22/30 Flush 23/30 Write block 415 len 4096 data: 13 24/30 Read block 10 len 4096 25/30 Read block 104 len 20480 26/30 Write block 82 len 24576 data: 10 7 11 14 13 9 27/30 Read block 127 len 8192 28/30 Read block 29 len 32768 29/30 Read block 230 len 24576 30/30 Write block 275 len 32768 data: 11 12 12 9 12 11 11 6 ---------------------------------------------------------------- Crucible gen:174 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 8 Sent Done Done false 1001 Acked Read 7 Sent Done Done false 1002 Acked Write 5 Sent Done Done false 1003 Acked Read 5 Sent Done Done false 1004 Acked Read 10 Sent Done Done false 1005 Acked Read 8 Sent Done Done false 1006 Acked Read 3 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Write 4 Sent Done Done false 1010 Acked Flush 0 Sent Done Done false 1011 Acked Write 8 Sent Done Done false 1012 Acked Write 9 Sent Done Done false 1013 Acked Read 5 Sent Done Done false 1014 Acked Write 8 Sent Done Done false 1015 Acked Read 3 Sent Done Done false 1016 Acked Write 1 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Read 2 Sent Done Done false 1019 Acked Flush 0 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Flush 0 Sent Done Done false 1022 Acked Write 1 Sent Done Done false 1023 Acked Read 1 Sent Done Done false 1024 Acked Read 5 Sent Done Done false 1025 Acked Write 6 Sent Done Done false 1026 Acked Read 2 Sent Done Done false 1027 Acked Read 8 Sent Done Done false 1028 Acked Read 6 Sent Done Done false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1021)) Some(JobId(1021)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:29.679429454Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:29.679641364Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1829,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 301 301 301 F F F 1 020-039 173 173 173 301 301 301 F F F 2 040-059 170 170 170 298 298 298 F F F 3 060-079 172 172 172 300 300 300 F F F 4 080-099 166 174 174 291 302 302 F T T <--- 5 100-119 173 174 174 301 302 302 F F F <--- 6 120-139 169 169 169 296 296 296 F F F 7 140-159 171 174 174 299 302 302 F F F <--- 8 160-179 172 174 174 300 303 303 F F F <--- 9 180-199 173 173 173 301 301 301 F F F 10 200-219 171 174 174 299 303 303 F F F <--- 11 220-239 173 173 173 301 301 301 F F F 12 240-259 170 170 170 297 297 297 F F F 13 260-279 172 172 172 300 300 300 F T T <--- 14 280-299 168 168 168 294 294 294 F T T <--- 15 300-319 170 170 170 297 297 297 F F F 16 320-339 170 174 174 298 304 304 F F F <--- 17 340-359 168 174 174 294 304 304 F F F <--- 18 360-379 164 174 174 287 303 303 F F F <--- 19 380-399 164 164 164 286 286 286 F F F 20 400-419 160 160 160 276 276 276 F T T <--- 21 420-439 170 174 174 298 303 303 F F F <--- 22 440-459 173 173 173 301 301 301 F F F 23 460-479 166 166 166 291 291 291 F F F 24 480-499 168 168 168 295 295 295 F F F 25 500-519 173 173 173 301 301 301 F F F 26 520-539 173 173 173 301 301 301 F F F 27 540-559 171 171 171 299 299 299 F F F 28 560-579 168 174 174 295 302 302 F F F <--- 29 580-599 161 174 174 278 303 303 F F F <--- Max gen: 174, Max flush: 304 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 87 Verifying data now No change after verify Loop: 87 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 301 301 301 F F F 1 020-039 173 173 173 301 301 301 F F F 2 040-059 170 170 170 298 298 298 F F F 3 060-079 172 172 172 300 300 300 F F F 4 080-099 175 175 175 305 305 305 F F F 5 100-119 174 174 174 302 302 302 F F F 6 120-139 169 169 169 296 296 296 F F F 7 140-159 174 174 174 302 302 302 F F F 8 160-179 174 174 174 303 303 303 F F F 9 180-199 173 173 173 301 301 301 F F F 10 200-219 174 174 174 303 303 303 F F F 11 220-239 173 173 173 301 301 301 F F F 12 240-259 170 170 170 297 297 297 F F F 13 260-279 175 175 175 305 305 305 F F F 14 280-299 175 175 175 305 305 305 F F F 15 300-319 170 170 170 297 297 297 F F F 16 320-339 174 174 174 304 304 304 F F F 17 340-359 174 174 174 304 304 304 F F F 18 360-379 174 174 174 303 303 303 F F F 19 380-399 164 164 164 286 286 286 F F F 20 400-419 175 175 175 305 305 305 F F F 21 420-439 174 174 174 303 303 303 F F F 22 440-459 173 173 173 301 301 301 F F F 23 460-479 166 166 166 291 291 291 F F F 24 480-499 168 168 168 295 295 295 F F F 25 500-519 173 173 173 301 301 301 F F F 26 520-539 173 173 173 301 301 301 F F F 27 540-559 171 171 171 299 299 299 F F F 28 560-579 174 174 174 302 302 302 F F F 29 580-599 174 174 174 303 303 303 F F F Max gen: 175, Max flush: 305 Begin loop 88 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.935561787Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:33.942684093Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.942724775Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.949453045Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.949712546Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.949742441Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.949750919Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.949762467Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837} {"msg":"Crucible 75f0bf2e-cdb0-4425-82d8-4694245b4424 has session id: 96348518-789b-4738-bbf9-d564ad74786c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950074079Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"Upstairs opts: Upstairs UUID: 75f0bf2e-cdb0-4425-82d8-4694245b4424, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950099804Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"Crucible stats registered with UUID: 75f0bf2e-cdb0-4425-82d8-4694245b4424","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950109782Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:33.95011868Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950259064Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"io task","client":"1","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950275761Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"io task","client":"0","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950285809Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"io task","client":"2","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950447119Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"task":"crutest"} {"msg":"The guest has requested activation with gen:176","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950470664Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837} {"msg":"75f0bf2e-cdb0-4425-82d8-4694245b4424 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950487121Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.95049595Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"client":"0","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950505418Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"client":"1","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950514826Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"client":"2","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950524294Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"io task","client":"2","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950534003Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"io task","client":"0","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950544101Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"io task","client":"1","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"ds_connection connected from Ok(127.0.0.1:37599)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950626625Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"io task","client":"0","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"ds_connection connected from Ok(127.0.0.1:53625)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950748472Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"io task","client":"1","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"ds_connection connected from Ok(127.0.0.1:48279)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.950774558Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"io task","client":"2","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.95129755Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"client":"1","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951309717Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"client":"2","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951321555Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"client":"0","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951592364Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951627808Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"[0]R flush_numbers[0..12]: [301, 301, 298, 300, 305, 302, 296, 302, 303, 301, 303, 301]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951709612Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"downstairs","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"[0]R generation[0..12]: [173, 173, 170, 172, 175, 174, 169, 174, 174, 173, 174, 173]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.95172086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"downstairs","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951731858Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"downstairs","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"[1]R flush_numbers[0..12]: [301, 301, 298, 300, 305, 302, 296, 302, 303, 301, 303, 301]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951742516Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"downstairs","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"[1]R generation[0..12]: [173, 173, 170, 172, 175, 174, 169, 174, 174, 173, 174, 173]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951753034Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"downstairs","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951763812Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"downstairs","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"[2]R flush_numbers[0..12]: [301, 301, 298, 300, 305, 302, 296, 302, 303, 301, 303, 301]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.95177662Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"downstairs","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"[2]R generation[0..12]: [173, 173, 170, 172, 175, 174, 169, 174, 174, 173, 174, 173]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951787198Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"downstairs","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951798066Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"downstairs","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"Max found gen is 176","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951807374Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"downstairs","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"Generation requested: 176 >= found:176","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951817112Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"downstairs","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"Next flush: 306","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951826281Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"downstairs","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951835539Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"":"downstairs","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951845047Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951853685Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"client":"0","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951863164Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"client":"1","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951872562Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"client":"2","":"downstairs","session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.95188229Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951891248Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"75f0bf2e-cdb0-4425-82d8-4694245b4424 is now active with session: 96348518-789b-4738-bbf9-d564ad74786c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951900967Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951909575Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"session_id":"96348518-789b-4738-bbf9-d564ad74786c"} {"msg":"The guest has finished waiting for activation with:176","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951918533Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837} {"msg":"Activated sub_volume 75f0bf2e-cdb0-4425-82d8-4694245b4424","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.951927742Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.9519357Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.952234854Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"task":"crutest"} 01/30 Write block 80 len 16384 data: 9 9 11 8 02/30 Read block 196 len 20480 03/30 Read block 545 len 40960 04/30 Write block 257 len 4096 data: 19 05/30 Write block 321 len 12288 data: 15 11 12 06/30 Read block 162 len 32768 07/30 Write block 246 len 20480 data: 17 18 17 17 14 08/30 Read block 39 len 32768 09/30 Write block 225 len 36864 data: 10 8 9 10 11 14 11 13 15 10/30 Write block 134 len 36864 data: 17 15 14 17 20 23 20 20 16 11/30 Write block 410 len 12288 data: 10 12 11 12/30 Read block 391 len 40960 13/30 Flush 14/30 Write block 182 len 16384 data: 15 15 14 14 15/30 Flush 16/30 Write block 290 len 8192 data: 15 15 17/30 Read block 346 len 28672 18/30 Write block 366 len 32768 data: 9 7 6 8 9 14 14 12 19/30 Write block 57 len 32768 data: 16 15 17 17 16 17 11 14 20/30 Read block 260 len 40960 21/30 Write block 164 len 4096 data: 9 22/30 Write block 385 len 12288 data: 10 12 11 23/30 Read block 451 len 36864 24/30 Write block 582 len 24576 data: 15 11 9 8 10 13 25/30 Write block 130 len 12288 data: 20 17 17 26/30 Write block 202 len 4096 data: 13 27/30 Write block 559 len 28672 data: 15 14 14 15 15 11 14 28/30 Write block 247 len 40960 data: 19 18 18 15 15 17 15 15 16 17 29/30 Write block 463 len 36864 data: 8 9 6 6 6 7 9 10 11 30/30 Read block 38 len 32768 ---------------------------------------------------------------- Crucible gen:176 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Done Done Done false 1001 Acked Read 5 Done Done Done false 1002 Acked Read 10 Done Done Done false 1003 Acked Write 1 Done Done Done false 1004 Acked Write 3 Done Done Done false 1005 Acked Read 8 Done Done Done false 1006 Acked Write 5 Sent Done Done false 1007 Acked Read 8 Sent Done Done false 1008 Acked Write 9 Sent Done Done false 1009 Acked Write 9 Sent Done Done false 1010 Acked Write 3 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Flush 0 Sent Done Done false 1013 Acked Write 4 Sent Done Done false 1014 Acked Flush 0 Sent Done Done false 1015 Acked Write 2 Sent Done Done false 1016 Acked Read 7 Sent Done Done false 1017 Acked Write 8 Sent Done Done false 1018 Acked Write 8 Sent Done Done false 1019 Acked Read 10 Sent Done Done false 1020 Acked Write 1 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Read 9 Sent Done Done false 1023 Acked Write 6 Sent Done Done false 1024 Acked Write 3 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Write 10 Sent Done Done false 1028 Acked Write 9 Sent Done Done false 1029 Acked Read 8 Sent Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 24 0 1 25 Done 6 30 29 65 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1014)) Some(JobId(1014)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:33.96037009Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:33.96058351Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1837,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 301 301 301 F F F 1 020-039 173 173 173 301 301 301 F F F 2 040-059 170 170 170 298 298 298 F T T <--- 3 060-079 172 172 172 300 300 300 F T T <--- 4 080-099 175 176 176 305 306 306 T F F <--- 5 100-119 174 174 174 302 302 302 F F F 6 120-139 169 176 176 296 306 306 F T T <--- 7 140-159 174 176 176 302 306 306 F F F <--- 8 160-179 174 174 174 303 303 303 F T T <--- 9 180-199 173 176 176 301 307 307 F F F <--- 10 200-219 174 174 174 303 303 303 F T T <--- 11 220-239 173 176 176 301 306 306 F F F <--- 12 240-259 170 176 176 297 306 306 T T T <--- 13 260-279 175 175 175 305 305 305 F F F 14 280-299 175 175 175 305 305 305 F T T <--- 15 300-319 170 170 170 297 297 297 F F F 16 320-339 174 176 176 304 306 306 T F F <--- 17 340-359 174 174 174 304 304 304 F F F 18 360-379 174 174 174 303 303 303 F T T <--- 19 380-399 164 164 164 286 286 286 F T T <--- 20 400-419 175 176 176 305 306 306 F F F <--- 21 420-439 174 174 174 303 303 303 F F F 22 440-459 173 173 173 301 301 301 F F F 23 460-479 166 166 166 291 291 291 F T T <--- 24 480-499 168 168 168 295 295 295 F F F 25 500-519 173 173 173 301 301 301 F F F 26 520-539 173 173 173 301 301 301 F F F 27 540-559 171 171 171 299 299 299 F T T <--- 28 560-579 174 174 174 302 302 302 F T T <--- 29 580-599 174 174 174 303 303 303 F T T <--- Max gen: 176, Max flush: 307 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 88 Verifying data now No change after verify Loop: 88 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 301 301 301 F F F 1 020-039 173 173 173 301 301 301 F F F 2 040-059 177 177 177 308 308 308 F F F 3 060-079 177 177 177 308 308 308 F F F 4 080-099 176 176 176 306 306 306 F F F 5 100-119 174 174 174 302 302 302 F F F 6 120-139 177 177 177 308 308 308 F F F 7 140-159 176 176 176 306 306 306 F F F 8 160-179 177 177 177 308 308 308 F F F 9 180-199 176 176 176 307 307 307 F F F 10 200-219 177 177 177 308 308 308 F F F 11 220-239 176 176 176 306 306 306 F F F 12 240-259 177 177 177 308 308 308 F F F 13 260-279 175 175 175 305 305 305 F F F 14 280-299 177 177 177 308 308 308 F F F 15 300-319 170 170 170 297 297 297 F F F 16 320-339 176 176 176 306 306 306 F F F 17 340-359 174 174 174 304 304 304 F F F 18 360-379 177 177 177 308 308 308 F F F 19 380-399 177 177 177 308 308 308 F F F 20 400-419 176 176 176 306 306 306 F F F 21 420-439 174 174 174 303 303 303 F F F 22 440-459 173 173 173 301 301 301 F F F 23 460-479 177 177 177 308 308 308 F F F 24 480-499 168 168 168 295 295 295 F F F 25 500-519 173 173 173 301 301 301 F F F 26 520-539 173 173 173 301 301 301 F F F 27 540-559 177 177 177 308 308 308 F F F 28 560-579 177 177 177 308 308 308 F F F 29 580-599 177 177 177 308 308 308 F F F Max gen: 177, Max flush: 308 Begin loop 89 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.231801258Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.240117041Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.240445839Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.240474154Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.240482762Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.24049411Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845} {"msg":"Crucible 15f128c5-b1d7-4d77-8153-5fe003fa3c8b has session id: fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.240856062Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"Upstairs opts: Upstairs UUID: 15f128c5-b1d7-4d77-8153-5fe003fa3c8b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.240885847Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"Crucible stats registered with UUID: 15f128c5-b1d7-4d77-8153-5fe003fa3c8b","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.240896645Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:38.240905933Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241048776Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"io task","client":"0","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241078891Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"io task","client":"1","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241089729Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"io task","client":"2","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241208026Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"task":"crutest"} {"msg":"The guest has requested activation with gen:178","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241225873Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845} {"msg":"15f128c5-b1d7-4d77-8153-5fe003fa3c8b active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241261636Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241271015Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"client":"0","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241280703Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"client":"1","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241290421Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"client":"2","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241299859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"io task","client":"2","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241309857Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"io task","client":"1","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"ds_connection connected from Ok(127.0.0.1:33543)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241360048Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"io task","client":"2","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241385153Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"io task","client":"0","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"ds_connection connected from Ok(127.0.0.1:55225)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241396571Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"io task","client":"1","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"ds_connection connected from Ok(127.0.0.1:39812)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.241512259Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"io task","client":"0","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242047459Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"client":"0","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242121415Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"client":"1","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242241223Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"client":"2","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242355322Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242434957Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"[0]R flush_numbers[0..12]: [301, 301, 308, 308, 306, 302, 308, 306, 308, 307, 308, 306]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242528759Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"downstairs","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"[0]R generation[0..12]: [173, 173, 177, 177, 176, 174, 177, 176, 177, 176, 177, 176]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242540037Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"downstairs","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242550945Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"downstairs","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"[1]R flush_numbers[0..12]: [301, 301, 308, 308, 306, 302, 308, 306, 308, 307, 308, 306]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242561453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"downstairs","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"[1]R generation[0..12]: [173, 173, 177, 177, 176, 174, 177, 176, 177, 176, 177, 176]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242571961Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"downstairs","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242582699Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"downstairs","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"[2]R flush_numbers[0..12]: [301, 301, 308, 308, 306, 302, 308, 306, 308, 307, 308, 306]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242593117Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"downstairs","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"[2]R generation[0..12]: [173, 173, 177, 177, 176, 174, 177, 176, 177, 176, 177, 176]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242603625Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"downstairs","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242614453Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"downstairs","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"Max found gen is 178","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242623741Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"downstairs","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"Generation requested: 178 >= found:178","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242635179Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"downstairs","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"Next flush: 309","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242644507Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"downstairs","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242653806Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"":"downstairs","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242663404Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242672152Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"client":"0","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242681621Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"client":"1","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242691129Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"client":"2","":"downstairs","session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242700817Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242709895Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"15f128c5-b1d7-4d77-8153-5fe003fa3c8b is now active with session: fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242719883Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible"Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 ,"level":30,"time":"2026-07-20T23:02:38.242735011Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"session_id":"fec5bd5b-3a2a-4c45-8a47-f97dfe77a1fb"} {"msg":"The guest has finished waiting for activation with:178","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242744129Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845} {"msg":"Activated sub_volume 15f128c5-b1d7-4d77-8153-5fe003fa3c8b","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242752117Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.242760156Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"task":"crutest"} Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.243000391Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"task":"crutest"} 01/30 Write block 138 len 8192 data: 21 24 02/30 Write block 328 len 8192 data: 11 13 03/30 Write block 31 len 40960 data: 25 23 21 17 16 14 15 16 17 16 04/30 Read block 270 len 12288 05/30 Read block 536 len 8192 06/30 Read block 359 len 4096 07/30 Write block 501 len 24576 data: 11 14 16 17 15 12 08/30 Write block 521 len 32768 data: 14 14 15 17 18 16 16 14 09/30 Write block 370 len 20480 data: 10 15 15 13 14 10/30 Flush 11/30 Write block 318 len 40960 data: 14 13 14 16 12 13 13 15 14 14 12/30 Read block 312 len 8192 13/30 Read block 260 len 8192 14/30 Write block 490 len 8192 data: 14 14 15/30 Write block 286 len 32768 data: 8 9 11 15 16 16 18 15 16/30 Flush 17/30 Flush 18/30 Write block 527 len 16384 data: 17 15 13 9 19/30 Write block 587 len 24576 data: 14 14 14 16 16 18 20/30 Read block 545 len 36864 21/30 Write block 549 len 28672 data: 13 13 14 17 15 16 14 22/30 Flush 23/30 Write block 49 len 36864 data: 10 8 9 8 8 11 16 19 17 24/30 Write block 443 len 40960 data: 15 13 13 11 10 10 11 12 12 12 25/30 Write block 464 len 28672 data: 10 7 7 7 8 10 11 26/30 Flush 27/30 Write block 162 len 40960 data: 8 7 10 11 11 10 8 8 12 12 28/30 Write block 348 len 40960 data: 8 10 10 13 11 11 12 11 13 15 29/30 Write block 539 len 24576 data: 7 8 9 7 10 10 30/30 Read block 382 len 16384 ---------------------------------------------------------------- Crucible gen:178 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 2 Done Done Done false 1001 Acked Write 2 Done Done Done false 1002 Acked Write 10 Done Done Done false 1003 Acked Read 3 Done Done Sent false 1004 Acked Read 2 Done Done Sent false 1005 Acked Read 1 Done Done Sent false 1006 Acked Write 6 Done Done Sent false 1007 Acked Write 8 Done Done Sent false 1008 Acked Write 5 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 10 Done Done Sent false 1011 Acked Read 2 Done Done Sent false 1012 Acked Read 2 Done Done Sent false 1013 Acked Write 2 Done Done Sent false 1014 Acked Write 8 Done Done Sent false 1015 Acked Flush 0 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 4 Done Done Sent false 1018 Acked Write 6 Done Done Sent false 1019 Acked Read 9 Done Done Sent false 1020 Acked Write 7 Done Done Sent false 1021 Acked Flush 0 Done Done Sent false 1022 Acked Write 9 Done Done Sent false 1023 Acked Write 10 Done Done Sent false 1024 Acked Write 7 Done Done Sent false 1025 Acked Flush 0 Done Done Sent false 1026 Acked Write 10 Done Done Sent false 1027 Acked Write 10 Done Done Sent false 1028 Acked Write 6 Done Done Sent false 1029 Acked Read 4 Done Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 1 27 28 Done 30 29 3 62 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1025)) Some(JobId(1025)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:38.251072269Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:38.251274851Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1845,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 301 301 301 F F F 1 020-039 178 178 173 309 309 301 F F T <--- 2 040-059 178 178 177 313 313 308 F F T <--- 3 060-079 177 177 177 308 308 308 F F F 4 080-099 176 176 176 306 306 306 F F F 5 100-119 174 174 174 302 302 302 F F F 6 120-139 178 178 177 309 309 308 F F T <--- 7 140-159 176 176 176 306 306 306 F F F 8 160-179 177 177 177 308 308 308 T T F <--- 9 180-199 176 176 176 307 307 307 F F F 10 200-219 177 177 177 308 308 308 F F F 11 220-239 176 176 176 306 306 306 F F F 12 240-259 177 177 177 308 308 308 F F F 13 260-279 175 175 175 305 305 305 F F F 14 280-299 178 178 177 310 310 308 F F F <--- 15 300-319 178 178 170 310 310 297 F F F <--- 16 320-339 178 178 176 310 310 306 F F T <--- 17 340-359 174 174 174 304 304 304 T T F <--- 18 360-379 178 178 177 309 309 308 F F F <--- 19 380-399 177 177 177 308 308 308 F F F 20 400-419 176 176 176 306 306 306 F F F 21 420-439 174 174 174 303 303 303 F F F 22 440-459 178 178 173 313 313 301 F F F <--- 23 460-479 178 178 177 313 313 308 F F F <--- 24 480-499 178 178 168 310 310 295 F F F <--- 25 500-519 178 178 173 309 309 301 F F F <--- 26 520-539 178 178 173 312 312 301 T T F <--- 27 540-559 178 178 177 312 312 308 T T F <--- 28 560-579 177 177 177 308 308 308 F F F 29 580-599 178 178 177 312 312 308 F F F <--- Max gen: 178, Max flush: 313 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 89 Verifying data now No change after verify Loop: 89 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 301 301 301 F F F 1 020-039 178 178 178 309 309 309 F F F 2 040-059 178 178 178 313 313 313 F F F 3 060-079 177 177 177 308 308 308 F F F 4 080-099 176 176 176 306 306 306 F F F 5 100-119 174 174 174 302 302 302 F F F 6 120-139 178 178 178 309 309 309 F F F 7 140-159 176 176 176 306 306 306 F F F 8 160-179 179 179 179 314 314 314 F F F 9 180-199 176 176 176 307 307 307 F F F 10 200-219 177 177 177 308 308 308 F F F 11 220-239 176 176 176 306 306 306 F F F 12 240-259 177 177 177 308 308 308 F F F 13 260-279 175 175 175 305 305 305 F F F 14 280-299 178 178 178 310 310 310 F F F 15 300-319 178 178 178 310 310 310 F F F 16 320-339 178 178 178 310 310 310 F F F 17 340-359 179 179 179 314 314 314 F F F 18 360-379 178 178 178 309 309 309 F F F 19 380-399 177 177 177 308 308 308 F F F 20 400-419 176 176 176 306 306 306 F F F 21 420-439 174 174 174 303 303 303 F F F 22 440-459 178 178 178 313 313 313 F F F 23 460-479 178 178 178 313 313 313 F F F 24 480-499 178 178 178 310 310 310 F F F 25 500-519 178 178 178 309 309 309 F F F 26 520-539 179 179 179 314 314 314 F F F 27 540-559 179 179 179 314 314 314 F F F 28 560-579 177 177 177 308 308 308 F F F 29 580-599 178 178 178 312 312 312 F F F Max gen: 179, Max flush: 314 Begin loop 90 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.508767189Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.516938429Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.517226755Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.51725401Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.517262818Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.517274146Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853} {"msg":"Crucible 83c4ca03-b4bd-4671-ad66-d32b02158d85 has session id: 81a32525-d2ae-4754-b527-6b0530512128","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.517636828Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"Upstairs opts: Upstairs UUID: 83c4ca03-b4bd-4671-ad66-d32b02158d85, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.517665423Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"Crucible stats registered with UUID: 83c4ca03-b4bd-4671-ad66-d32b02158d85","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.517676691Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:42.517685789Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.517819484Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"io task","client":"1","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.517838001Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"io task","client":"0","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.517847809Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"io task","client":"2","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.518009688Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"task":"crutest"} {"msg":"The guest has requested activation with gen:180","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.518021766Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853} {"msg":"83c4ca03-b4bd-4671-ad66-d32b02158d85 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.51805576Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.518065208Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"client":"0","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.518074856Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"client":"1","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.518084474Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"client":"2","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.518093983Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"io task","client":"2","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.518103911Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"io task","client":"0","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.518113629Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"io task","client":"1","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"ds_connection connected from Ok(127.0.0.1:38734)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.51816359Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"io task","client":"0","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"ds_connection connected from Ok(127.0.0.1:34320)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.518228217Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"io task","client":"1","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"ds_connection connected from Ok(127.0.0.1:59261)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.518302384Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"io task","client":"2","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.518880685Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"client":"1","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.518895742Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"client":"2","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.51890644Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"client":"0","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.51923101Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519254375Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"[0]R flush_numbers[0..12]: [301, 309, 313, 308, 306, 302, 309, 306, 314, 307, 308, 306]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519267563Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"downstairs","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"[0]R generation[0..12]: [173, 178, 178, 177, 176, 174, 178, 176, 179, 176, 177, 176]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519278791Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"downstairs","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519289929Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"downstairs","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"[1]R flush_numbers[0..12]: [301, 309, 313, 308, 306, 302, 309, 306, 314, 307, 308, 306]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519300637Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"downstairs","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"[1]R generation[0..12]: [173, 178, 178, 177, 176, 174, 178, 176, 179, 176, 177, 176]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519311365Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"downstairs","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519322303Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"downstairs","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"[2]R flush_numbers[0..12]: [301, 309, 313, 308, 306, 302, 309, 306, 314, 307, 308, 306]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519333021Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"downstairs","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"[2]R generation[0..12]: [173, 178, 178, 177, 176, 174, 178, 176, 179, 176, 177, 176]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519343699Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"downstairs","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519354597Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"downstairs","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"Max found gen is 180","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519363945Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"downstairs","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"Generation requested: 180 >= found:180","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519373643Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"downstairs","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"Next flush: 315","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519383081Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"downstairs","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519392539Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"":"downstairs","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519402188Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519412276Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"client":"0","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519421894Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"client":"1","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519431472Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"client":"2","":"downstairs","session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.51944123Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519450309Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"83c4ca03-b4bd-4671-ad66-d32b02158d85 is now active with session: 81a32525-d2ae-4754-b527-6b0530512128","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519460267Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519468995Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"session_id":"81a32525-d2ae-4754-b527-6b0530512128"} {"msg":"The guest has finished waiting for activation with:180","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519478123Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853} {"msg":"Activated sub_volume 83c4ca03-b4bd-4671-ad66-d32b02158d85","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519486162Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.51949412Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.519831317Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"task":"crutest"} 01/30 Flush 02/30 Flush 03/30 Write block 202 len 36864 data: 14 13 12 12 12 13 13 19 23 04/30 Read block 205 len 4096 05/30 Read block 221 len 24576 06/30 Write block 566 len 36864 data: 13 13 10 10 10 10 9 11 10 07/30 Read block 195 len 20480 08/30 Read block 142 len 36864 09/30 Read block 409 len 12288 10/30 Flush 11/30 Write block 494 len 4096 data: 13 12/30 Read block 537 len 32768 13/30 Read block 273 len 32768 14/30 Read block 206 len 28672 15/30 Read block 96 len 36864 16/30 Write block 522 len 24576 data: 15 16 18 19 17 18 17/30 Read block 401 len 4096 18/30 Write block 536 len 12288 data: 8 7 6 19/30 Write block 86 len 32768 data: 14 10 10 12 11 10 9 12 20/30 Read block 230 len 8192 21/30 Read block 408 len 36864 22/30 Read block 323 len 12288 23/30 Write block 454 len 8192 data: 11 10 24/30 Read block 394 len 36864 25/30 Read block 14 len 4096 26/30 Read block 110 len 28672 27/30 Read block 25 len 8192 28/30 Write block 268 len 20480 data: 11 14 11 9 8 29/30 Write block 556 len 4096 data: 14 30/30 Read block 51 len 8192 ---------------------------------------------------------------- Crucible gen:180 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Flush 0 Done Done Sent false 1001 Acked Flush 0 Done Done Sent false 1002 Acked Write 9 Done Done Sent false 1003 Acked Read 1 Done Done Sent false 1004 Acked Read 6 Done Done Sent false 1005 Acked Write 9 Done Done Sent false 1006 Acked Read 5 Done Done Sent false 1007 Acked Read 9 Done Done Sent false 1008 Acked Read 3 Done Done Sent false 1009 Acked Flush 0 Done Done Sent false 1010 Acked Write 1 Done Done Sent false 1011 Acked Read 8 Done Done Sent false 1012 Acked Read 8 Done Done Sent false 1013 Acked Read 7 Done Done Sent false 1014 Acked Read 9 Done Done Sent false 1015 Acked Write 6 Done Done Sent false 1016 Acked Read 1 Done Done Sent false 1017 Acked Write 3 Done Done Sent false 1018 Acked Write 8 Done Done Sent false 1019 Acked Read 2 Done Done Sent false 1020 Acked Read 9 Done Done Sent false 1021 Acked Read 3 Done Done Sent false 1022 Acked Write 2 Done Done Sent false 1023 Acked Read 9 Done Done Sent false 1024 Acked Read 1 Done Done Sent false 1025 Acked Read 7 Done Done Sent false 1026 Acked Read 2 Done Done Sent false 1027 Acked Write 5 Done Done Sent false 1028 Acked Write 1 Done Done Sent false 1029 Acked Read 2 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1009)) Some(JobId(1009)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:42.528592506Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:42.52878451Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1853,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 301 301 301 F F F 1 020-039 178 178 178 309 309 309 F F F 2 040-059 178 178 178 313 313 313 F F F 3 060-079 177 177 177 308 308 308 F F F 4 080-099 176 176 176 306 306 306 T T F <--- 5 100-119 174 174 174 302 302 302 F F F 6 120-139 178 178 178 309 309 309 F F F 7 140-159 176 176 176 306 306 306 F F F 8 160-179 179 179 179 314 314 314 F F F 9 180-199 176 176 176 307 307 307 F F F 10 200-219 180 180 177 317 317 308 F F F <--- 11 220-239 176 176 176 306 306 306 F F F 12 240-259 177 177 177 308 308 308 F F F 13 260-279 175 175 175 305 305 305 T T F <--- 14 280-299 178 178 178 310 310 310 F F F 15 300-319 178 178 178 310 310 310 F F F 16 320-339 178 178 178 310 310 310 F F F 17 340-359 179 179 179 314 314 314 F F F 18 360-379 178 178 178 309 309 309 F F F 19 380-399 177 177 177 308 308 308 F F F 20 400-419 176 176 176 306 306 306 F F F 21 420-439 174 174 174 303 303 303 F F F 22 440-459 178 178 178 313 313 313 T T F <--- 23 460-479 178 178 178 313 313 313 F F F 24 480-499 178 178 178 310 310 310 T T F <--- 25 500-519 178 178 178 309 309 309 F F F 26 520-539 179 179 179 314 314 314 T T F <--- 27 540-559 179 179 179 314 314 314 T T F <--- 28 560-579 180 180 177 317 317 308 F F F <--- 29 580-599 178 178 178 312 312 312 F F F Max gen: 180, Max flush: 317 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 90 Verifying data now No change after verify Loop: 90 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 301 301 301 F F F 1 020-039 178 178 178 309 309 309 F F F 2 040-059 178 178 178 313 313 313 F F F 3 060-079 177 177 177 308 308 308 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 174 174 174 302 302 302 F F F 6 120-139 178 178 178 309 309 309 F F F 7 140-159 176 176 176 306 306 306 F F F 8 160-179 179 179 179 314 314 314 F F F 9 180-199 176 176 176 307 307 307 F F F 10 200-219 180 180 180 317 317 317 F F F 11 220-239 176 176 176 306 306 306 F F F 12 240-259 177 177 177 308 308 308 F F F 13 260-279 181 181 181 318 318 318 F F F 14 280-299 178 178 178 310 310 310 F F F 15 300-319 178 178 178 310 310 310 F F F 16 320-339 178 178 178 310 310 310 F F F 17 340-359 179 179 179 314 314 314 F F F 18 360-379 178 178 178 309 309 309 F F F 19 380-399 177 177 177 308 308 308 F F F 20 400-419 176 176 176 306 306 306 F F F 21 420-439 174 174 174 303 303 303 F F F 22 440-459 181 181 181 318 318 318 F F F 23 460-479 178 178 178 313 313 313 F F F 24 480-499 181 181 181 318 318 318 F F F 25 500-519 178 178 178 309 309 309 F F F 26 520-539 181 181 181 318 318 318 F F F 27 540-559 181 181 181 318 318 318 F F F 28 560-579 180 180 180 317 317 317 F F F 29 580-599 178 178 178 312 312 312 F F F Max gen: 181, Max flush: 318 Begin loop 91 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.768533182Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.776649342Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.776941728Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.776970762Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.776979251Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.776992688Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861} {"msg":"Crucible 55d14173-4ba8-40ef-979d-cb888f9c04b5 has session id: 0db4c41c-00d5-416a-a6ae-104b781d3b81","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777367978Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"Upstairs opts: Upstairs UUID: 55d14173-4ba8-40ef-979d-cb888f9c04b5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777398752Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"Crucible stats registered with UUID: 55d14173-4ba8-40ef-979d-cb888f9c04b5","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.77740996Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:46.777419518Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777562391Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"io task","client":"1","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777595725Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"io task","client":"0","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777606893Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"io task","client":"2","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777744277Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"task":"crutest"} {"msg":"The guest has requested activation with gen:182","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777760044Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861} {"msg":"55d14173-4ba8-40ef-979d-cb888f9c04b5 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777795378Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777804926Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"client":"0","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777814564Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"client":"1","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777824142Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"client":"2","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777833621Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"io task","client":"2","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777845988Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"io task","client":"0","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777856036Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"io task","client":"1","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"ds_connection connected from Ok(127.0.0.1:53428)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777938161Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"io task","client":"0","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"ds_connection connected from Ok(127.0.0.1:47513)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777967336Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"io task","client":"2","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"ds_connection connected from Ok(127.0.0.1:34792)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.777978783Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"io task","client":"1","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.778620763Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"client":"1","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.77863566Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"client":"0","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.778755958Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"client":"2","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.778966139Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.778975767Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"[0]R flush_numbers[0..12]: [301, 309, 313, 308, 318, 302, 309, 306, 314, 307, 317, 306]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779052452Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"downstairs","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"[0]R generation[0..12]: [173, 178, 178, 177, 181, 174, 178, 176, 179, 176, 180, 176]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.77906374Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"downstairs","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779074688Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"downstairs","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"[1]R flush_numbers[0..12]: [301, 309, 313, 308, 318, 302, 309, 306, 314, 307, 317, 306]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779085246Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"downstairs","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"[1]R generation[0..12]: [173, 178, 178, 177, 181, 174, 178, 176, 179, 176, 180, 176]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779095534Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"downstairs","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779108532Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"downstairs","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"[2]R flush_numbers[0..12]: [301, 309, 313, 308, 318, 302, 309, 306, 314, 307, 317, 306]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.77911913Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"downstairs","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"[2]R generation[0..12]: [173, 178, 178, 177, 181, 174, 178, 176, 179, 176, 180, 176]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779129558Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"downstairs","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779140346Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"downstairs","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"Max found gen is 182","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779149764Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"downstairs","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"Generation requested: 182 >= found:182","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779159452Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"downstairs","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"Next flush: 319","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779168581Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"downstairs","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779177869Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"":"downstairs","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779187897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779196665Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"client":"0","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779206593Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"client":"1","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779216612Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"client":"2","":"downstairs","session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.7792266Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779236128Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"55d14173-4ba8-40ef-979d-cb888f9c04b5 is now active with session: 0db4c41c-00d5-416a-a6ae-104b781d3b81","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779246026Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779256274Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"session_id":"0db4c41c-00d5-416a-a6ae-104b781d3b81"} {"msg":"The guest has finished waiting for activation with:182","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779265512Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861} {"msg":"Activated sub_volume 55d14173-4ba8-40ef-979d-cb888f9c04b5","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779273471Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779281519Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.779631744Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"task":"crutest"} 01/30 Write block 142 len 12288 data: 17 14 14 02/30 Write block 232 len 28672 data: 14 16 15 13 13 13 12 03/30 Read block 237 len 12288 04/30 Read block 81 len 36864 05/30 Read block 573 len 8192 06/30 Write block 22 len 28672 data: 14 15 16 18 15 12 15 07/30 Flush 08/30 Write block 545 len 36864 data: 14 13 12 12 14 14 15 18 16 09/30 Read block 209 len 32768 10/30 Write block 175 len 36864 data: 17 15 16 19 17 14 12 16 16 11/30 Read block 554 len 32768 12/30 Read block 138 len 40960 13/30 Write block 204 len 32768 data: 13 13 13 14 14 20 24 20 14/30 Write block 385 len 40960 data: 11 13 12 12 10 12 13 14 17 14 15/30 Read block 135 len 8192 16/30 Write block 175 len 16384 data: 18 16 17 20 17/30 Flush 18/30 Write block 537 len 16384 data: 8 7 8 9 19/30 Read block 509 len 20480 20/30 Read block 525 len 4096 21/30 Flush 22/30 Write block 171 len 4096 data: 13 23/30 Read block 338 len 40960 24/30 Read block 129 len 12288 25/30 Read block 5 len 28672 26/30 Write block 190 len 28672 data: 14 19 21 25 23 23 23 27/30 Write block 438 len 28672 data: 12 13 16 17 17 16 14 28/30 Write block 193 len 40960 data: 26 24 24 24 19 17 13 15 12 15 29/30 Write block 576 len 32768 data: 12 14 15 14 14 14 16 12 30/30 Write block 113 len 20480 data: 18 17 14 10 10 ---------------------------------------------------------------- Crucible gen:182 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Write 7 Done Done Done false 1002 Acked Read 3 Done Done Done false 1003 Acked Read 9 Done Done Sent false 1004 Acked Read 2 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Flush 0 Done Done Sent false 1007 Acked Write 9 Done Done Sent false 1008 Acked Read 8 Done Done Sent false 1009 Acked Write 9 Done Done Sent false 1010 Acked Read 8 Done Done Sent false 1011 Acked Read 10 Done Done Sent false 1012 Acked Write 8 Done Done Sent false 1013 Acked Write 10 Done Done Sent false 1014 Acked Read 2 Done Done Sent false 1015 Acked Write 4 Done Done Sent false 1016 Acked Flush 0 Done Done Sent false 1017 Acked Write 4 Done Done Sent false 1018 Acked Read 5 Done Done Sent false 1019 Acked Read 1 Done Done Sent false 1020 Acked Flush 0 Done Done Sent false 1021 Acked Write 1 Done Done Sent false 1022 Acked Read 10 Done Done Sent false 1023 Acked Read 3 Done Done Sent false 1024 Acked Read 7 Done Done Sent false 1025 Acked Write 7 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Write 10 Sent Sent Sent false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 3 3 27 33 Done 27 27 3 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1020)) Some(JobId(1020)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:46.788565191Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:46.788792808Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1861,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 301 301 301 F F F 1 020-039 182 182 178 319 319 309 F F F <--- 2 040-059 178 178 178 313 313 313 F F F 3 060-079 177 177 177 308 308 308 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 174 174 174 302 302 302 T T F <--- 6 120-139 178 178 178 309 309 309 F F F 7 140-159 182 182 176 319 319 306 F F T <--- 8 160-179 182 182 179 320 320 314 T T F <--- 9 180-199 182 182 176 320 320 307 T T F <--- 10 200-219 182 182 180 320 320 317 T T F <--- 11 220-239 182 182 176 319 319 306 F F T <--- 12 240-259 177 177 177 308 308 308 F F F 13 260-279 181 181 181 318 318 318 F F F 14 280-299 178 178 178 310 310 310 F F F 15 300-319 178 178 178 310 310 310 F F F 16 320-339 178 178 178 310 310 310 F F F 17 340-359 179 179 179 314 314 314 F F F 18 360-379 178 178 178 309 309 309 F F F 19 380-399 182 182 177 320 320 308 F F F <--- 20 400-419 176 176 176 306 306 306 F F F 21 420-439 174 174 174 303 303 303 T T F <--- 22 440-459 181 181 181 318 318 318 T T F <--- 23 460-479 178 178 178 313 313 313 F F F 24 480-499 181 181 181 318 318 318 F F F 25 500-519 178 178 178 309 309 309 F F F 26 520-539 182 182 181 321 321 318 F F F <--- 27 540-559 182 182 181 321 321 318 F F F <--- 28 560-579 180 180 180 317 317 317 T T F <--- 29 580-599 178 178 178 312 312 312 T T F <--- Max gen: 182, Max flush: 321 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 91 Verifying data now No change after verify Loop: 91 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 301 301 301 F F F 1 020-039 182 182 182 319 319 319 F F F 2 040-059 178 178 178 313 313 313 F F F 3 060-079 177 177 177 308 308 308 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 183 183 183 322 322 322 F F F 6 120-139 178 178 178 309 309 309 F F F 7 140-159 182 182 182 319 319 319 F F F 8 160-179 183 183 183 322 322 322 F F F 9 180-199 183 183 183 322 322 322 F F F 10 200-219 183 183 183 322 322 322 F F F 11 220-239 182 182 182 319 319 319 F F F 12 240-259 177 177 177 308 308 308 F F F 13 260-279 181 181 181 318 318 318 F F F 14 280-299 178 178 178 310 310 310 F F F 15 300-319 178 178 178 310 310 310 F F F 16 320-339 178 178 178 310 310 310 F F F 17 340-359 179 179 179 314 314 314 F F F 18 360-379 178 178 178 309 309 309 F F F 19 380-399 182 182 182 320 320 320 F F F 20 400-419 176 176 176 306 306 306 F F F 21 420-439 183 183 183 322 322 322 F F F 22 440-459 183 183 183 322 322 322 F F F 23 460-479 178 178 178 313 313 313 F F F 24 480-499 181 181 181 318 318 318 F F F 25 500-519 178 178 178 309 309 309 F F F 26 520-539 182 182 182 321 321 321 F F F 27 540-559 182 182 182 321 321 321 F F F 28 560-579 183 183 183 322 322 322 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 183, Max flush: 322 Begin loop 92 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.045717252Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:51.052810094Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.052860814Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.059591504Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.059859733Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.059888488Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.059896897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.059908364Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869} {"msg":"Crucible ed4de390-fe99-47c9-90d3-778ecffbcaf4 has session id: 7703dfc6-74ba-4236-abef-b01bb03da60c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060265388Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"Upstairs opts: Upstairs UUID: ed4de390-fe99-47c9-90d3-778ecffbcaf4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060297551Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"Crucible stats registered with UUID: ed4de390-fe99-47c9-90d3-778ecffbcaf4","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060308699Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:51.060317708Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060444074Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"io task","client":"1","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060476588Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"io task","client":"0","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060487696Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"io task","client":"2","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060621641Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"task":"crutest"} {"msg":"The guest has requested activation with gen:184","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060645076Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869} {"msg":"ed4de390-fe99-47c9-90d3-778ecffbcaf4 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060657774Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060666802Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"client":"0","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.06067709Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"client":"1","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060687039Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"client":"2","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060696727Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"io task","client":"2","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060706875Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"io task","client":"0","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060716923Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"io task","client":"1","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"ds_connection connected from Ok(127.0.0.1:37303)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060780231Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"io task","client":"2","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"ds_connection connected from Ok(127.0.0.1:54585)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060798388Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"io task","client":"1","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"ds_connection connected from Ok(127.0.0.1:38730)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.060813935Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"io task","client":"0","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.061523682Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"client":"1","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.061543728Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"client":"2","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.061590549Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"client":"0","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.061817787Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.06185297Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"[0]R flush_numbers[0..12]: [301, 319, 313, 308, 318, 322, 309, 319, 322, 322, 322, 319]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.061947203Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"downstairs","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"[0]R generation[0..12]: [173, 182, 178, 177, 181, 183, 178, 182, 183, 183, 183, 182]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.06195869Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"downstairs","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.061969818Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"downstairs","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"[1]R flush_numbers[0..12]: [301, 319, 313, 308, 318, 322, 309, 319, 322, 322, 322, 319]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.061980296Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"downstairs","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"[1]R generation[0..12]: [173, 182, 178, 177, 181, 183, 178, 182, 183, 183, 183, 182]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.061990804Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"downstairs","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062001892Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"downstairs","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"[2]R flush_numbers[0..12]: [301, 319, 313, 308, 318, 322, 309, 319, 322, 322, 322, 319]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.06201254Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"downstairs","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"[2]R generation[0..12]: [173, 182, 178, 177, 181, 183, 178, 182, 183, 183, 183, 182]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062025278Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"downstairs","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062036086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"downstairs","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"Max found gen is 184","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062045494Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"downstairs","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"Generation requested: 184 >= found:184","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062055272Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"downstairs","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"Next flush: 323","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062064601Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"downstairs","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062073969Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"":"downstairs","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062083687Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062092515Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"client":"0","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062102283Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"client":"1","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062111762Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"client":"2","":"downstairs","session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.06212159Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062130658Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"ed4de390-fe99-47c9-90d3-778ecffbcaf4 is now active with session: 7703dfc6-74ba-4236-abef-b01bb03da60c","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062140546Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062149225Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"session_id":"7703dfc6-74ba-4236-abef-b01bb03da60c"} {"msg":"The guest has finished waiting for activation with:184","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062158243Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869} {"msg":"Activated sub_volume ed4de390-fe99-47c9-90d3-778ecffbcaf4","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062166292Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.06217589Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.062514516Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"task":"crutest"} 01/30 Write block 316 len 12288 data: 17 18 15 02/30 Read block 157 len 20480 03/30 Read block 138 len 28672 04/30 Read block 295 len 24576 05/30 Read block 497 len 16384 06/30 Read block 281 len 12288 07/30 Write block 235 len 4096 data: 14 08/30 Write block 531 len 28672 data: 8 8 8 9 8 9 9 09/30 Read block 566 len 16384 10/30 Read block 381 len 32768 11/30 Read block 424 len 20480 12/30 Write block 360 len 8192 data: 12 7 13/30 Write block 198 len 40960 data: 18 14 16 13 16 14 14 14 14 15 14/30 Write block 15 len 12288 data: 19 19 15 15/30 Read block 36 len 32768 16/30 Write block 490 len 16384 data: 15 15 13 13 17/30 Read block 327 len 12288 18/30 Write block 123 len 28672 data: 15 14 14 17 18 18 18 19/30 Write block 202 len 20480 data: 17 15 15 15 15 20/30 Read block 35 len 36864 21/30 Read block 70 len 8192 22/30 Write block 551 len 12288 data: 16 19 17 23/30 Read block 539 len 12288 24/30 Write block 444 len 28672 data: 15 14 12 11 11 12 13 25/30 Write block 8 len 24576 data: 15 16 17 22 21 19 26/30 Read block 89 len 28672 27/30 Write block 468 len 4096 data: 9 28/30 Read block 460 len 4096 29/30 Write block 30 len 32768 data: 24 26 24 22 18 17 15 16 30/30 Write block 216 len 32768 data: 18 18 16 12 14 13 11 10 ---------------------------------------------------------------- Crucible gen:184 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Done false 1001 Acked Read 5 Sent Done Done false 1002 Acked Read 7 Sent Done Done false 1003 Acked Read 6 Sent Done Done false 1004 Acked Read 4 Sent Done Done false 1005 Acked Read 3 Sent Done Done false 1006 Acked Write 1 Sent Done Done false 1007 Acked Write 7 Sent Done Done false 1008 Acked Read 4 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Read 5 Sent Done Done false 1011 Acked Write 2 Sent Done Done false 1012 Acked Write 10 Sent Done Done false 1013 Acked Write 3 Sent Done Done false 1014 Acked Read 8 Sent Done Done false 1015 Acked Write 4 Sent Done Done false 1016 Acked Read 3 Sent Done Done false 1017 Acked Write 7 Sent Done Done false 1018 Acked Write 5 Sent Done Done false 1019 Acked Read 9 Sent Done Done false 1020 Acked Read 2 Sent Done Done false 1021 Acked Write 3 Sent Done Done false 1022 Acked Read 3 Sent Done Done false 1023 Acked Write 7 Sent Done Done false 1024 Acked Write 6 Sent Done Done false 1025 Acked Read 7 Sent Done Done false 1026 Acked Write 1 Sent Done Done false 1027 Acked Read 1 Sent Done Done false 1028 Acked Write 8 Sent Sent Sent false 1029 Acked Write 8 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 2 2 33 Done 1 28 28 57 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None None None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:51.070873341Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:51.07109158Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1869,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 173 173 173 301 301 301 F T T <--- 1 020-039 182 182 182 319 319 319 F T T <--- 2 040-059 178 178 178 313 313 313 F F F 3 060-079 177 177 177 308 308 308 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 183 183 183 322 322 322 F F F 6 120-139 178 178 178 309 309 309 F T T <--- 7 140-159 182 182 182 319 319 319 F F F 8 160-179 183 183 183 322 322 322 F F F 9 180-199 183 183 183 322 322 322 F T T <--- 10 200-219 183 183 183 322 322 322 F T T <--- 11 220-239 182 182 182 319 319 319 F T T <--- 12 240-259 177 177 177 308 308 308 F F F 13 260-279 181 181 181 318 318 318 F F F 14 280-299 178 178 178 310 310 310 F F F 15 300-319 178 178 178 310 310 310 T T T 16 320-339 178 178 178 310 310 310 F F F 17 340-359 179 179 179 314 314 314 F F F 18 360-379 178 178 178 309 309 309 F T T <--- 19 380-399 182 182 182 320 320 320 F F F 20 400-419 176 176 176 306 306 306 F F F 21 420-439 183 183 183 322 322 322 F F F 22 440-459 183 183 183 322 322 322 F T T <--- 23 460-479 178 178 178 313 313 313 F T T <--- 24 480-499 181 181 181 318 318 318 F T T <--- 25 500-519 178 178 178 309 309 309 F F F 26 520-539 182 182 182 321 321 321 F T T <--- 27 540-559 182 182 182 321 321 321 F T T <--- 28 560-579 183 183 183 322 322 322 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 183, Max flush: 322 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 92 Verifying data now No change after verify Loop: 92 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 184 184 184 323 323 323 F F F 1 020-039 184 184 184 323 323 323 F F F 2 040-059 178 178 178 313 313 313 F F F 3 060-079 177 177 177 308 308 308 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 183 183 183 322 322 322 F F F 6 120-139 184 184 184 323 323 323 F F F 7 140-159 182 182 182 319 319 319 F F F 8 160-179 183 183 183 322 322 322 F F F 9 180-199 184 184 184 323 323 323 F F F 10 200-219 184 184 184 323 323 323 F F F 11 220-239 184 184 184 323 323 323 F F F 12 240-259 177 177 177 308 308 308 F F F 13 260-279 181 181 181 318 318 318 F F F 14 280-299 178 178 178 310 310 310 F F F 15 300-319 184 184 184 323 323 323 F F F 16 320-339 178 178 178 310 310 310 F F F 17 340-359 179 179 179 314 314 314 F F F 18 360-379 184 184 184 323 323 323 F F F 19 380-399 182 182 182 320 320 320 F F F 20 400-419 176 176 176 306 306 306 F F F 21 420-439 183 183 183 322 322 322 F F F 22 440-459 184 184 184 323 323 323 F F F 23 460-479 184 184 184 323 323 323 F F F 24 480-499 184 184 184 323 323 323 F F F 25 500-519 178 178 178 309 309 309 F F F 26 520-539 184 184 184 323 323 323 F F F 27 540-559 184 184 184 323 323 323 F F F 28 560-579 183 183 183 322 322 322 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 184, Max flush: 323 Begin loop 93 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.337397068Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:55.344476412Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.344527563Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.351193544Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.351459245Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.351487549Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.351496008Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.351507446Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877} {"msg":"Crucible d78f3420-00c2-4d35-ba82-89d169a6d3e9 has session id: c4f4293e-e2d0-485f-a542-9f3850be2478","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.351813228Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"Upstairs opts: Upstairs UUID: d78f3420-00c2-4d35-ba82-89d169a6d3e9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.35185939Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"Crucible stats registered with UUID: d78f3420-00c2-4d35-ba82-89d169a6d3e9","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.351870168Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:55.351881146Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352030378Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"io task","client":"1","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352046345Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"io task","client":"0","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352056373Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"io task","client":"2","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352210774Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"task":"crutest"} {"msg":"The guest has requested activation with gen:186","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352227341Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877} {"msg":"d78f3420-00c2-4d35-ba82-89d169a6d3e9 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352263684Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352273152Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"client":"0","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.35228271Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"client":"1","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352292269Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"client":"2","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352301747Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"io task","client":"2","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352311665Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"io task","client":"0","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352321573Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"io task","client":"1","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"ds_connection connected from Ok(127.0.0.1:38603)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352361636Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"io task","client":"2","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"ds_connection connected from Ok(127.0.0.1:41607)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352372544Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"io task","client":"1","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"ds_connection connected from Ok(127.0.0.1:62555)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.352486292Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"io task","client":"0","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353056915Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"client":"2","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353080581Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"client":"1","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353116894Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"client":"0","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353374546Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353384754Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"[0]R flush_numbers[0..12]: [323, 323, 313, 308, 318, 322, 323, 319, 322, 323, 323, 323]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353422847Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"downstairs","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"[0]R generation[0..12]: [184, 184, 178, 177, 181, 183, 184, 182, 183, 184, 184, 184]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353434155Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"downstairs","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353445263Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"downstairs","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"[1]R flush_numbers[0..12]: [323, 323, 313, 308, 318, 322, 323, 319, 322, 323, 323, 323]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353456031Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"downstairs","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"[1]R generation[0..12]: [184, 184, 178, 177, 181, 183, 184, 182, 183, 184, 184, 184]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353466669Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"downstairs","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353477627Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"downstairs","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"[2]R flush_numbers[0..12]: [323, 323, 313, 308, 318, 322, 323, 319, 322, 323, 323, 323]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353488155Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"downstairs","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"[2]R generation[0..12]: [184, 184, 178, 177, 181, 183, 184, 182, 183, 184, 184, 184]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353498903Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"downstairs","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353509791Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"downstairs","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"Max found gen is 185","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353521638Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"downstairs","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"Generation requested: 186 >= found:185","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353531496Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"downstairs","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"Next flush: 324","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353540805Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"downstairs","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353550233Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"":"downstairs","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353559961Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.35356859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"client":"0","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353578448Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"client":"1","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353588236Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"client":"2","":"downstairs","session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353597994Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353607082Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"d78f3420-00c2-4d35-ba82-89d169a6d3e9 is now active with session: c4f4293e-e2d0-485f-a542-9f3850be2478","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.3536169Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353625779Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"session_id":"c4f4293e-e2d0-485f-a542-9f3850be2478"} {"msg":"The guest has finished waiting for activation with:186","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353634837Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877} {"msg":"Activated sub_volume d78f3420-00c2-4d35-ba82-89d169a6d3e9","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353642786Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.353650784Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.35399115Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"task":"crutest"} 01/30 Write block 390 len 16384 data: 13 14 15 18 02/30 Write block 2 len 16384 data: 5 6 8 8 03/30 Read block 68 len 16384 04/30 Write block 492 len 20480 data: 14 14 14 12 9 05/30 Read block 181 len 36864 06/30 Read block 339 len 4096 07/30 Read block 475 len 28672 08/30 Read block 596 len 16384 09/30 Write block 278 len 36864 data: 10 13 12 12 7 5 5 8 9 10/30 Read block 565 len 32768 11/30 Read block 167 len 40960 12/30 Write block 437 len 20480 data: 15 13 14 17 18 13/30 Read block 193 len 32768 14/30 Write block 361 len 40960 data: 8 8 9 10 11 10 8 7 9 11 15/30 Write block 536 len 16384 data: 10 10 8 9 16/30 Read block 249 len 8192 17/30 Read block 120 len 24576 18/30 Flush 19/30 Write block 575 len 12288 data: 11 13 15 20/30 Read block 404 len 20480 21/30 Write block 483 len 4096 data: 11 22/30 Write block 495 len 16384 data: 13 10 11 13 23/30 Write block 18 len 8192 data: 17 17 24/30 Read block 354 len 20480 25/30 Read block 431 len 12288 26/30 Write block 355 len 16384 data: 12 14 16 11 27/30 Write block 120 len 28672 data: 11 13 11 16 15 15 18 28/30 Read block 444 len 24576 29/30 Read block 335 len 36864 30/30 Write block 198 len 20480 data: 19 15 17 14 18 ---------------------------------------------------------------- Crucible gen:186 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 4 Sent Done Done false 1001 Acked Write 4 Sent Done Done false 1002 Acked Read 4 Sent Done Done false 1003 Acked Write 5 Sent Done Done false 1004 Acked Read 9 Sent Done Done false 1005 Acked Read 1 Sent Done Done false 1006 Acked Read 7 Sent Done Done false 1007 Acked Read 4 Sent Done Done false 1008 Acked Write 9 Sent Done Done false 1009 Acked Read 8 Sent Done Done false 1010 Acked Read 10 Sent Done Done false 1011 Acked Write 5 Sent Done Done false 1012 Acked Read 8 Sent Done Done false 1013 Acked Write 10 Sent Done Done false 1014 Acked Write 4 Sent Done Done false 1015 Acked Read 2 Sent Done Done false 1016 Acked Read 6 Sent Done Done false 1017 Acked Flush 0 Sent Done Done false 1018 Acked Write 3 Sent Done Done false 1019 Acked Read 5 Sent Done Done false 1020 Acked Write 1 Sent Done Done false 1021 Acked Write 4 Sent Done Done false 1022 Acked Write 2 Sent Done Done false 1023 Acked Read 5 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Write 4 Sent Done Done false 1026 Acked Write 7 Sent Done Done false 1027 Acked Read 6 Sent Done Done false 1028 Acked Read 9 Sent Done Done false 1029 Acked Write 5 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 30 1 1 32 Done 0 29 29 58 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1017)) Some(JobId(1017)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:55.362753039Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:55.362918229Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1877,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 184 186 186 323 324 324 F T T <--- 1 020-039 184 184 184 323 323 323 F F F 2 040-059 178 178 178 313 313 313 F F F 3 060-079 177 177 177 308 308 308 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 183 183 183 322 322 322 F F F 6 120-139 184 184 184 323 323 323 F T T <--- 7 140-159 182 182 182 319 319 319 F F F 8 160-179 183 183 183 322 322 322 F F F 9 180-199 184 184 184 323 323 323 F T T <--- 10 200-219 184 184 184 323 323 323 F T T <--- 11 220-239 184 184 184 323 323 323 F F F 12 240-259 177 177 177 308 308 308 F F F 13 260-279 181 186 186 318 324 324 F F F <--- 14 280-299 178 186 186 310 324 324 F F F <--- 15 300-319 184 184 184 323 323 323 F F F 16 320-339 178 178 178 310 310 310 F F F 17 340-359 179 179 179 314 314 314 F T T <--- 18 360-379 184 186 186 323 324 324 F F F <--- 19 380-399 182 186 186 320 324 324 F F F <--- 20 400-419 176 176 176 306 306 306 F F F 21 420-439 183 186 186 322 324 324 F F F <--- 22 440-459 184 186 186 323 324 324 F F F <--- 23 460-479 184 184 184 323 323 323 F F F 24 480-499 184 186 186 323 324 324 F T T <--- 25 500-519 178 178 178 309 309 309 F F F 26 520-539 184 186 186 323 324 324 F F F <--- 27 540-559 184 184 184 323 323 323 F F F 28 560-579 183 183 183 322 322 322 F T T <--- 29 580-599 183 183 183 322 322 322 F F F Max gen: 186, Max flush: 324 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 93 Verifying data now No change after verify Loop: 93 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 325 325 325 F F F 1 020-039 184 184 184 323 323 323 F F F 2 040-059 178 178 178 313 313 313 F F F 3 060-079 177 177 177 308 308 308 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 183 183 183 322 322 322 F F F 6 120-139 187 187 187 325 325 325 F F F 7 140-159 182 182 182 319 319 319 F F F 8 160-179 183 183 183 322 322 322 F F F 9 180-199 187 187 187 325 325 325 F F F 10 200-219 187 187 187 325 325 325 F F F 11 220-239 184 184 184 323 323 323 F F F 12 240-259 177 177 177 308 308 308 F F F 13 260-279 186 186 186 324 324 324 F F F 14 280-299 186 186 186 324 324 324 F F F 15 300-319 184 184 184 323 323 323 F F F 16 320-339 178 178 178 310 310 310 F F F 17 340-359 187 187 187 325 325 325 F F F 18 360-379 186 186 186 324 324 324 F F F 19 380-399 186 186 186 324 324 324 F F F 20 400-419 176 176 176 306 306 306 F F F 21 420-439 186 186 186 324 324 324 F F F 22 440-459 186 186 186 324 324 324 F F F 23 460-479 184 184 184 323 323 323 F F F 24 480-499 187 187 187 325 325 325 F F F 25 500-519 178 178 178 309 309 309 F F F 26 520-539 186 186 186 324 324 324 F F F 27 540-559 184 184 184 323 323 323 F F F 28 560-579 187 187 187 325 325 325 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 187, Max flush: 325 Begin loop 94 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.624182551Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:59.631326533Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.631376833Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.63817545Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.6384406Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.638469385Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.638478203Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.638489791Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885} {"msg":"Crucible 04f940a2-b298-4b77-b339-2faa5b477650 has session id: 5883b9e0-a317-4c2e-a387-e8751a8963cd","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.638822279Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"Upstairs opts: Upstairs UUID: 04f940a2-b298-4b77-b339-2faa5b477650, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.638845624Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"Crucible stats registered with UUID: 04f940a2-b298-4b77-b339-2faa5b477650","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.638855443Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:59.638864551Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.639013883Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"io task","client":"0","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.63903059Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"io task","client":"1","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.639042688Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"io task","client":"2","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.639214085Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"task":"crutest"} {"msg":"The guest has requested activation with gen:188","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.639230622Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885} {"msg":"04f940a2-b298-4b77-b339-2faa5b477650 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.639247179Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.639255758Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"client":"0","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.639265436Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"client":"1","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.639275014Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"client":"2","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.639284442Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"io task","client":"2","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.63929459Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"io task","client":"0","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.639304429Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"io task","client":"1","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"ds_connection connected from Ok(127.0.0.1:33989)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.6393507Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"io task","client":"2","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"ds_connection connected from Ok(127.0.0.1:45661)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.639361798Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"io task","client":"1","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"ds_connection connected from Ok(127.0.0.1:52817)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.639468978Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"io task","client":"0","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640049219Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"client":"1","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640075364Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"client":"2","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640147221Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"client":"0","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640369039Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640386346Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"[0]R flush_numbers[0..12]: [325, 323, 313, 308, 318, 322, 325, 319, 322, 325, 325, 323]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640443845Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"downstairs","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"[0]R generation[0..12]: [187, 184, 178, 177, 181, 183, 187, 182, 183, 187, 187, 184]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640455013Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"downstairs","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640465811Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"downstairs","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"[1]R flush_numbers[0..12]: [325, 323, 313, 308, 318, 322, 325, 319, 322, 325, 325, 323]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640476519Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"downstairs","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"[1]R generation[0..12]: [187, 184, 178, 177, 181, 183, 187, 182, 183, 187, 187, 184]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640486907Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"downstairs","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640497735Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"downstairs","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"[2]R flush_numbers[0..12]: [325, 323, 313, 308, 318, 322, 325, 319, 322, 325, 325, 323]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640508173Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"downstairs","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"[2]R generation[0..12]: [187, 184, 178, 177, 181, 183, 187, 182, 183, 187, 187, 184]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640518631Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"downstairs","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640529419Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"downstairs","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"Max found gen is 188","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640538917Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"downstairs","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"Generation requested: 188 >= found:188","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640548716Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"downstairs","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"Next flush: 326","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640558074Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"downstairs","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640568922Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"":"downstairs","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.64057854Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640587348Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"client":"0","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640597076Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"client":"1","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640606665Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"client":"2","":"downstairs","session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640616373Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640625361Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"04f940a2-b298-4b77-b339-2faa5b477650 is now active with session: 5883b9e0-a317-4c2e-a387-e8751a8963cd","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640635179Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640643878Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"session_id":"5883b9e0-a317-4c2e-a387-e8751a8963cd"} {"msg":"The guest has finished waiting for activation with:188","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640653086Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885} {"msg":"Activated sub_volume 04f940a2-b298-4b77-b339-2faa5b477650","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640661034Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.640669083Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.641011589Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"task":"crutest"} 01/30 Write block 361 len 36864 data: 9 9 10 11 12 11 9 8 10 02/30 Flush 03/30 Write block 451 len 20480 data: 13 13 12 12 11 04/30 Write block 456 len 16384 data: 15 13 13 12 05/30 Read block 120 len 32768 06/30 Read block 29 len 16384 07/30 Read block 356 len 8192 08/30 Write block 566 len 28672 data: 14 14 11 11 11 11 10 09/30 Read block 222 len 24576 10/30 Write block 173 len 8192 data: 14 19 11/30 Read block 559 len 28672 12/30 Read block 272 len 40960 13/30 Write block 128 len 28672 data: 19 19 21 18 18 18 18 14/30 Read block 182 len 32768 15/30 Read block 185 len 24576 16/30 Read block 174 len 40960 17/30 Write block 273 len 32768 data: 7 11 12 13 13 11 14 13 18/30 Write block 330 len 16384 data: 17 16 18 14 19/30 Write block 383 len 36864 data: 17 14 12 14 13 13 11 14 15 20/30 Read block 175 len 24576 21/30 Write block 406 len 12288 data: 13 10 9 22/30 Write block 528 len 32768 data: 16 14 10 9 9 9 10 9 23/30 Write block 222 len 8192 data: 12 11 24/30 Write block 146 len 12288 data: 10 12 8 25/30 Read block 587 len 12288 26/30 Write block 539 len 4096 data: 10 27/30 Read block 495 len 24576 28/30 Read block 416 len 28672 29/30 Read block 348 len 12288 30/30 Read block 337 len 28672 ---------------------------------------------------------------- Crucible gen:188 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 9 Done Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Write 5 Sent Done Done false 1003 Acked Write 4 Sent Done Done false 1004 Acked Read 8 Sent Done Done false 1005 Acked Read 4 Sent Done Done false 1006 Acked Read 2 Sent Done Done false 1007 Acked Write 7 Sent Done Done false 1008 Acked Read 6 Sent Done Done false 1009 Acked Write 2 Sent Done Done false 1010 Acked Read 7 Sent Done Done false 1011 Acked Read 10 Sent Done Done false 1012 Acked Write 7 Sent Done Done false 1013 Acked Read 8 Sent Done Done false 1014 Acked Read 6 Sent Done Done false 1015 Acked Read 10 Sent Done Done false 1016 Acked Write 8 Sent Done Done false 1017 Acked Write 4 Sent Done Done false 1018 Acked Write 9 Sent Done Done false 1019 Acked Read 6 Sent Done Done false 1020 Acked Write 3 Sent Done Done false 1021 Acked Write 8 Sent Done Done false 1022 Acked Write 2 Sent Done Done false 1023 Acked Write 3 Sent Done Done false 1024 Acked Read 3 Sent Done Done false 1025 Acked Write 1 Sent Done Done false 1026 Acked Read 6 Sent Done Done false 1027 Acked Read 7 Sent Done Done false 1028 Acked Read 3 Sent Done Done false 1029 Acked Read 7 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 0 30 Done 1 29 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1001)) Some(JobId(1001)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:02:59.649556828Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:02:59.649749582Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1885,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 325 325 325 F F F 1 020-039 184 184 184 323 323 323 F F F 2 040-059 178 178 178 313 313 313 F F F 3 060-079 177 177 177 308 308 308 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 183 183 183 322 322 322 F F F 6 120-139 187 187 187 325 325 325 F T T <--- 7 140-159 182 182 182 319 319 319 F T T <--- 8 160-179 183 183 183 322 322 322 F T T <--- 9 180-199 187 187 187 325 325 325 F F F 10 200-219 187 187 187 325 325 325 F F F 11 220-239 184 184 184 323 323 323 F T T <--- 12 240-259 177 177 177 308 308 308 F F F 13 260-279 186 186 186 324 324 324 F T T <--- 14 280-299 186 186 186 324 324 324 F T T <--- 15 300-319 184 184 184 323 323 323 F F F 16 320-339 178 178 178 310 310 310 F T T <--- 17 340-359 187 187 187 325 325 325 F F F 18 360-379 186 188 188 324 326 326 T F F <--- 19 380-399 186 186 186 324 324 324 F T T <--- 20 400-419 176 176 176 306 306 306 F T T <--- 21 420-439 186 186 186 324 324 324 F F F 22 440-459 186 186 186 324 324 324 F T T <--- 23 460-479 184 184 184 323 323 323 F F F 24 480-499 187 187 187 325 325 325 F F F 25 500-519 178 178 178 309 309 309 F F F 26 520-539 186 186 186 324 324 324 F T T <--- 27 540-559 184 184 184 323 323 323 F F F 28 560-579 187 187 187 325 325 325 F T T <--- 29 580-599 183 183 183 322 322 322 F F F Max gen: 188, Max flush: 326 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 94 Verifying data now No change after verify Loop: 94 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 325 325 325 F F F 1 020-039 184 184 184 323 323 323 F F F 2 040-059 178 178 178 313 313 313 F F F 3 060-079 177 177 177 308 308 308 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 183 183 183 322 322 322 F F F 6 120-139 189 189 189 327 327 327 F F F 7 140-159 189 189 189 327 327 327 F F F 8 160-179 189 189 189 327 327 327 F F F 9 180-199 187 187 187 325 325 325 F F F 10 200-219 187 187 187 325 325 325 F F F 11 220-239 189 189 189 327 327 327 F F F 12 240-259 177 177 177 308 308 308 F F F 13 260-279 189 189 189 327 327 327 F F F 14 280-299 189 189 189 327 327 327 F F F 15 300-319 184 184 184 323 323 323 F F F 16 320-339 189 189 189 327 327 327 F F F 17 340-359 187 187 187 325 325 325 F F F 18 360-379 188 188 188 326 326 326 F F F 19 380-399 189 189 189 327 327 327 F F F 20 400-419 189 189 189 327 327 327 F F F 21 420-439 186 186 186 324 324 324 F F F 22 440-459 189 189 189 327 327 327 F F F 23 460-479 184 184 184 323 323 323 F F F 24 480-499 187 187 187 325 325 325 F F F 25 500-519 178 178 178 309 309 309 F F F 26 520-539 189 189 189 327 327 327 F F F 27 540-559 184 184 184 323 323 323 F F F 28 560-579 189 189 189 327 327 327 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 189, Max flush: 327 Begin loop 95 Downstairs to restart: 2 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.910039806Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.918413928Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.91872143Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.918749755Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.918758204Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.918769711Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893} {"msg":"Crucible f4c37436-2404-4955-b4b5-ce2a55ca8c66 has session id: 1b25d0af-617f-484a-be4e-951832b1c01b","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919085872Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"Upstairs opts: Upstairs UUID: f4c37436-2404-4955-b4b5-ce2a55ca8c66, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919114417Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"Crucible stats registered with UUID: f4c37436-2404-4955-b4b5-ce2a55ca8c66","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919125465Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:03:03.919134433Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919285635Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"io task","client":"0","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919316049Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"io task","client":"1","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919326937Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"io task","client":"2","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919460052Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"task":"crutest"} {"msg":"The guest has requested activation with gen:190","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919486707Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893} {"msg":"f4c37436-2404-4955-b4b5-ce2a55ca8c66 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919499425Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919508343Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"client":"0","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919519931Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"client":"1","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919529709Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"client":"2","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919539257Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"io task","client":"2","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919568482Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"io task","client":"0","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919591468Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"io task","client":"1","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"ds_connection connected from Ok(127.0.0.1:38681)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919602765Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"io task","client":"1","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"ds_connection connected from Ok(127.0.0.1:43129)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919617493Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"io task","client":"0","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"ds_connection connected from Ok(127.0.0.1:47400)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.919735701Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"io task","client":"2","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.92032377Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"client":"0","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920337828Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"client":"1","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920348936Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"client":"2","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920628713Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920663987Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"[0]R flush_numbers[0..12]: [325, 323, 313, 308, 318, 322, 327, 327, 327, 325, 325, 327]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.92075126Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"downstairs","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"[0]R generation[0..12]: [187, 184, 178, 177, 181, 183, 189, 189, 189, 187, 187, 189]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920762778Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"downstairs","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920776506Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"downstairs","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"[1]R flush_numbers[0..12]: [325, 323, 313, 308, 318, 322, 327, 327, 327, 325, 325, 327]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920787224Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"downstairs","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"[1]R generation[0..12]: [187, 184, 178, 177, 181, 183, 189, 189, 189, 187, 187, 189]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920797642Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"downstairs","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.92080846Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"downstairs","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"[2]R flush_numbers[0..12]: [325, 323, 313, 308, 318, 322, 327, 327, 327, 325, 325, 327]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920818898Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"downstairs","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"[2]R generation[0..12]: [187, 184, 178, 177, 181, 183, 189, 189, 189, 187, 187, 189]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920829406Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"downstairs","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920840284Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"downstairs","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"Max found gen is 190","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920849712Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"downstairs","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"Generation requested: 190 >= found:190","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.92085946Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"downstairs","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"Next flush: 328","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920868818Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"downstairs","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920878117Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"":"downstairs","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920887845Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920896613Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"client":"0","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920906201Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"client":"1","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920915789Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"client":"2","":"downstairs","session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920926817Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920935736Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"f4c37436-2404-4955-b4b5-ce2a55ca8c66 is now active with session: 1b25d0af-617f-484a-be4e-951832b1c01b","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920945484Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920954192Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"session_id":"1b25d0af-617f-484a-be4e-951832b1c01b"} {"msg":"The guest has finished waiting for activation with:190","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920963261Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893} {"msg":"Activated sub_volume f4c37436-2404-4955-b4b5-ce2a55ca8c66","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920971169Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.920979138Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.921333121Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"task":"crutest"} 01/30 Write block 430 len 12288 data: 12 9 8 02/30 Write block 308 len 12288 data: 16 15 18 03/30 Read block 388 len 16384 04/30 Read block 142 len 28672 05/30 Write block 141 len 24576 data: 21 18 15 15 12 11 06/30 Write block 336 len 28672 data: 17 18 20 17 15 10 13 07/30 Read block 473 len 40960 08/30 Write block 408 len 16384 data: 10 6 11 13 09/30 Write block 510 len 4096 data: 10 10/30 Write block 513 len 12288 data: 13 16 18 11/30 Flush 12/30 Write block 469 len 16384 data: 11 12 12 8 13/30 Flush 14/30 Write block 292 len 12288 data: 19 16 14 15/30 Write block 208 len 16384 data: 15 21 25 21 16/30 Read block 504 len 4096 17/30 Write block 425 len 28672 data: 9 11 13 12 13 13 10 18/30 Read block 360 len 40960 19/30 Read block 79 len 16384 20/30 Write block 39 len 16384 data: 18 17 18 19 21/30 Read block 24 len 12288 22/30 Read block 405 len 4096 23/30 Write block 365 len 12288 data: 13 12 10 24/30 Read block 125 len 36864 25/30 Read block 3 len 36864 26/30 Write block 407 len 8192 data: 11 11 27/30 Write block 494 len 28672 data: 15 14 11 12 14 14 14 28/30 Write block 401 len 12288 data: 12 13 13 29/30 Read block 55 len 8192 30/30 Read block 175 len 36864 ---------------------------------------------------------------- Crucible gen:190 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 3 Done Done Sent false 1001 Acked Write 3 Done Done Sent false 1002 Acked Read 4 Done Done Sent false 1003 Acked Read 7 Done Done Sent false 1004 Acked Write 6 Done Done Sent false 1005 Acked Write 7 Done Done Sent false 1006 Acked Read 10 Done Done Sent false 1007 Acked Write 4 Done Done Sent false 1008 Acked Write 1 Done Done Sent false 1009 Acked Write 3 Done Done Sent false 1010 Acked Flush 0 Done Done Sent false 1011 Acked Write 4 Done Done Sent false 1012 Acked Flush 0 Done Done Sent false 1013 Acked Write 3 Done Done Sent false 1014 Acked Write 4 Done Done Sent false 1015 Acked Read 1 Done Done Sent false 1016 Acked Write 7 Done Done Sent false 1017 Acked Read 10 Done Done Sent false 1018 Acked Read 4 Done Done Sent false 1019 Acked Write 4 Done Done Sent false 1020 Acked Read 3 Done Done Sent false 1021 Acked Read 1 Done Done Sent false 1022 Acked Write 3 Done Done Sent false 1023 Acked Read 9 Done Done Sent false 1024 Acked Read 9 Done Done Sent false 1025 Acked Write 2 Done Done Sent false 1026 Acked Write 7 Done Done Sent false 1027 Acked Write 3 Done Done Sent false 1028 Acked Read 2 Done Done Sent false 1029 Acked Read 9 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 30 30 Done 30 30 0 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1012)) Some(JobId(1012)) None Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:03:03.929268085Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:03.929451331Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1893,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 325 325 325 F F F 1 020-039 184 184 184 323 323 323 T T F <--- 2 040-059 178 178 178 313 313 313 T T F <--- 3 060-079 177 177 177 308 308 308 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 183 183 183 322 322 322 F F F 6 120-139 189 189 189 327 327 327 F F F 7 140-159 190 190 189 328 328 327 F F F <--- 8 160-179 189 189 189 327 327 327 F F F 9 180-199 187 187 187 325 325 325 F F F 10 200-219 187 187 187 325 325 325 T T F <--- 11 220-239 189 189 189 327 327 327 F F F 12 240-259 177 177 177 308 308 308 F F F 13 260-279 189 189 189 327 327 327 F F F 14 280-299 189 189 189 327 327 327 T T F <--- 15 300-319 190 190 184 328 328 323 F F F <--- 16 320-339 190 190 189 328 328 327 F F F <--- 17 340-359 190 190 187 328 328 325 F F F <--- 18 360-379 188 188 188 326 326 326 T T F <--- 19 380-399 189 189 189 327 327 327 F F F 20 400-419 190 190 189 328 328 327 T T F <--- 21 420-439 190 190 186 328 328 324 T T F <--- 22 440-459 189 189 189 327 327 327 F F F 23 460-479 190 190 184 329 329 323 F F F <--- 24 480-499 187 187 187 325 325 325 T T F <--- 25 500-519 190 190 178 328 328 309 T T F <--- 26 520-539 189 189 189 327 327 327 F F F 27 540-559 184 184 184 323 323 323 F F F 28 560-579 189 189 189 327 327 327 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 190, Max flush: 329 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 95 Verifying data now No change after verify Loop: 95 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 325 325 325 F F F 1 020-039 191 191 191 330 330 330 F F F 2 040-059 191 191 191 330 330 330 F F F 3 060-079 177 177 177 308 308 308 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 183 183 183 322 322 322 F F F 6 120-139 189 189 189 327 327 327 F F F 7 140-159 190 190 190 328 328 328 F F F 8 160-179 189 189 189 327 327 327 F F F 9 180-199 187 187 187 325 325 325 F F F 10 200-219 191 191 191 330 330 330 F F F 11 220-239 189 189 189 327 327 327 F F F 12 240-259 177 177 177 308 308 308 F F F 13 260-279 189 189 189 327 327 327 F F F 14 280-299 191 191 191 330 330 330 F F F 15 300-319 190 190 190 328 328 328 F F F 16 320-339 190 190 190 328 328 328 F F F 17 340-359 190 190 190 328 328 328 F F F 18 360-379 191 191 191 330 330 330 F F F 19 380-399 189 189 189 327 327 327 F F F 20 400-419 191 191 191 330 330 330 F F F 21 420-439 191 191 191 330 330 330 F F F 22 440-459 189 189 189 327 327 327 F F F 23 460-479 190 190 190 329 329 329 F F F 24 480-499 191 191 191 330 330 330 F F F 25 500-519 191 191 191 330 330 330 F F F 26 520-539 189 189 189 327 327 327 F F F 27 540-559 184 184 184 323 323 323 F F F 28 560-579 189 189 189 327 327 327 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 191, Max flush: 330 Begin loop 96 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.185401838Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:03:08.192507597Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.192561737Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.199242626Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.199511116Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.19953917Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.199547639Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.199561336Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901} {"msg":"Crucible 756b2306-cb98-4eac-87e5-a9d6809c56e9 has session id: 48e30d89-066a-44f3-bb97-f2e2a5d3d29e","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.199931417Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"Upstairs opts: Upstairs UUID: 756b2306-cb98-4eac-87e5-a9d6809c56e9, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.199960822Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"Crucible stats registered with UUID: 756b2306-cb98-4eac-87e5-a9d6809c56e9","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.199971799Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:03:08.199980978Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.200108224Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"io task","client":"2","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.200125961Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"io task","client":"0","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.200136139Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"io task","client":"1","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.200305577Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"task":"crutest"} {"msg":"The guest has requested activation with gen:192","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.200322364Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901} {"msg":"756b2306-cb98-4eac-87e5-a9d6809c56e9 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.20034527Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.200354738Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"client":"0","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.200364276Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"client":"1","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.200373884Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"client":"2","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.200383422Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"io task","client":"2","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.20039469Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"io task","client":"0","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.200404498Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"io task","client":"1","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"ds_connection connected from Ok(127.0.0.1:54376)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.200463857Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"io task","client":"0","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"ds_connection connected from Ok(127.0.0.1:46403)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.200474905Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"io task","client":"2","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"ds_connection connected from Ok(127.0.0.1:44533)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.20050324Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"io task","client":"1","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.20119682Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"client":"1","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201214917Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"client":"2","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201226345Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"client":"0","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201523529Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201558992Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"[0]R flush_numbers[0..12]: [325, 330, 330, 308, 318, 322, 327, 328, 327, 325, 330, 327]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201640567Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"downstairs","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"[0]R generation[0..12]: [187, 191, 191, 177, 181, 183, 189, 190, 189, 187, 191, 189]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201651955Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"downstairs","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201663053Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"downstairs","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"[1]R flush_numbers[0..12]: [325, 330, 330, 308, 318, 322, 327, 328, 327, 325, 330, 327]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201673591Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"downstairs","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"[1]R generation[0..12]: [187, 191, 191, 177, 181, 183, 189, 190, 189, 187, 191, 189]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201684209Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"downstairs","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201697056Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"downstairs","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"[2]R flush_numbers[0..12]: [325, 330, 330, 308, 318, 322, 327, 328, 327, 325, 330, 327]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201707714Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"downstairs","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"[2]R generation[0..12]: [187, 191, 191, 177, 181, 183, 189, 190, 189, 187, 191, 189]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201718262Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"downstairs","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.20172908Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"downstairs","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"Max found gen is 192","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201738449Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"downstairs","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"Generation requested: 192 >= found:192","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201748237Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"downstairs","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"Next flush: 331","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201757525Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"downstairs","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201766873Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"":"downstairs","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201776631Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.20178545Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"client":"0","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201795188Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"client":"1","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201804736Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"client":"2","":"downstairs","session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201814684Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201823823Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"756b2306-cb98-4eac-87e5-a9d6809c56e9 is now active with session: 48e30d89-066a-44f3-bb97-f2e2a5d3d29e","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201833711Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201843799Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"session_id":"48e30d89-066a-44f3-bb97-f2e2a5d3d29e"} {"msg":"The guest has finished waiting for activation with:192","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201852957Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901} {"msg":"Activated sub_volume 756b2306-cb98-4eac-87e5-a9d6809c56e9","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201860946Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.201868964Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.202226047Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"task":"crutest"} 01/30 Write block 238 len 28672 data: 13 8 13 13 14 16 15 02/30 Flush 03/30 Flush 04/30 Write block 427 len 16384 data: 14 13 14 14 05/30 Read block 50 len 20480 06/30 Flush 07/30 Flush 08/30 Flush 09/30 Write block 36 len 32768 data: 16 17 17 19 18 19 20 20 10/30 Read block 349 len 20480 11/30 Read block 385 len 40960 12/30 Read block 505 len 16384 13/30 Write block 186 len 40960 data: 15 14 13 16 15 20 22 27 25 25 14/30 Read block 261 len 4096 15/30 Write block 427 len 4096 data: 15 16/30 Read block 370 len 16384 17/30 Flush 18/30 Write block 455 len 4096 data: 12 19/30 Read block 43 len 28672 20/30 Read block 117 len 12288 21/30 Read block 532 len 28672 22/30 Read block 570 len 16384 23/30 Write block 265 len 40960 data: 9 10 13 12 15 12 10 9 8 12 24/30 Write block 73 len 4096 data: 6 25/30 Write block 367 len 4096 data: 11 26/30 Read block 386 len 36864 27/30 Read block 99 len 4096 28/30 Read block 203 len 40960 29/30 Read block 447 len 8192 30/30 Read block 44 len 24576 ---------------------------------------------------------------- Crucible gen:192 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Flush 0 Sent Done Done false 1002 Acked Flush 0 Sent Done Done false 1003 Acked Write 4 Sent Done Done false 1004 Acked Read 5 Sent Done Done false 1005 Acked Flush 0 Sent Done Done false 1006 Acked Flush 0 Sent Done Done false 1007 Acked Flush 0 Sent Done Done false 1008 Acked Write 8 Sent Done Done false 1009 Acked Read 5 Sent Done Done false 1010 Acked Read 10 Sent Done Done false 1011 Acked Read 4 Sent Done Done false 1012 Acked Write 10 Sent Done Done false 1013 Acked Read 1 Sent Done Done false 1014 Acked Write 1 Sent Done Done false 1015 Acked Read 4 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 1 Sent Done Done false 1018 Acked Read 7 Sent Done Done false 1019 Acked Read 3 Sent Done Done false 1020 Acked Read 7 Sent Done Done false 1021 Acked Read 4 Sent Done Done false 1022 Acked Write 10 Sent Done Done false 1023 Acked Write 1 Sent Done Done false 1024 Acked Write 1 Sent Done Done false 1025 Acked Read 9 Sent Done Done false 1026 Acked Read 1 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Read 2 Sent Done Done false 1029 Acked Read 6 Sent Done Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 0 0 29 Done 1 30 30 61 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:03:08.211440262Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:08.211628256Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1901,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 325 325 325 F F F 1 020-039 191 192 192 330 336 336 F F F <--- 2 040-059 191 192 192 330 336 336 F F F <--- 3 060-079 177 177 177 308 308 308 F T T <--- 4 080-099 181 181 181 318 318 318 F F F 5 100-119 183 183 183 322 322 322 F F F 6 120-139 189 189 189 327 327 327 F F F 7 140-159 190 190 190 328 328 328 F F F 8 160-179 189 189 189 327 327 327 F F F 9 180-199 187 192 192 325 336 336 F F F <--- 10 200-219 191 191 191 330 330 330 F F F 11 220-239 189 192 192 327 331 331 T F F <--- 12 240-259 177 192 192 308 331 331 T F F <--- 13 260-279 189 189 189 327 327 327 F T T <--- 14 280-299 191 191 191 330 330 330 F F F 15 300-319 190 190 190 328 328 328 F F F 16 320-339 190 190 190 328 328 328 F F F 17 340-359 190 190 190 328 328 328 F F F 18 360-379 191 191 191 330 330 330 F T T <--- 19 380-399 189 189 189 327 327 327 F F F 20 400-419 191 191 191 330 330 330 F F F 21 420-439 191 192 192 330 336 336 F F F <--- 22 440-459 189 189 189 327 327 327 F T T <--- 23 460-479 190 190 190 329 329 329 F F F 24 480-499 191 191 191 330 330 330 F F F 25 500-519 191 191 191 330 330 330 F F F 26 520-539 189 189 189 327 327 327 F F F 27 540-559 184 184 184 323 323 323 F F F 28 560-579 189 189 189 327 327 327 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 192, Max flush: 336 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 96 Verifying data now No change after verify Loop: 96 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 325 325 325 F F F 1 020-039 192 192 192 336 336 336 F F F 2 040-059 192 192 192 336 336 336 F F F 3 060-079 193 193 193 337 337 337 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 183 183 183 322 322 322 F F F 6 120-139 189 189 189 327 327 327 F F F 7 140-159 190 190 190 328 328 328 F F F 8 160-179 189 189 189 327 327 327 F F F 9 180-199 192 192 192 336 336 336 F F F 10 200-219 191 191 191 330 330 330 F F F 11 220-239 192 192 192 331 331 331 F F F 12 240-259 192 192 192 331 331 331 F F F 13 260-279 193 193 193 337 337 337 F F F 14 280-299 191 191 191 330 330 330 F F F 15 300-319 190 190 190 328 328 328 F F F 16 320-339 190 190 190 328 328 328 F F F 17 340-359 190 190 190 328 328 328 F F F 18 360-379 193 193 193 337 337 337 F F F 19 380-399 189 189 189 327 327 327 F F F 20 400-419 191 191 191 330 330 330 F F F 21 420-439 192 192 192 336 336 336 F F F 22 440-459 193 193 193 337 337 337 F F F 23 460-479 190 190 190 329 329 329 F F F 24 480-499 191 191 191 330 330 330 F F F 25 500-519 191 191 191 330 330 330 F F F 26 520-539 189 189 189 327 327 327 F F F 27 540-559 184 184 184 323 323 323 F F F 28 560-579 189 189 189 327 327 327 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 193, Max flush: 337 Begin loop 97 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.466305922Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.474508776Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.474811009Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.474846243Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.474855761Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.474867429Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909} {"msg":"Crucible 167fe7e2-cb36-4480-b1be-67d136c24da0 has session id: 9ce1a9e7-5c21-4708-b4d2-dec756e7637e","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475172801Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"Upstairs opts: Upstairs UUID: 167fe7e2-cb36-4480-b1be-67d136c24da0, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.47523459Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"Crucible stats registered with UUID: 167fe7e2-cb36-4480-b1be-67d136c24da0","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475246808Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:03:12.475257785Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.47539199Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"io task","client":"1","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475408597Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"io task","client":"0","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475418675Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"io task","client":"2","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475579815Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"task":"crutest"} {"msg":"The guest has requested activation with gen:194","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475603701Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909} {"msg":"167fe7e2-cb36-4480-b1be-67d136c24da0 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475619828Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475628546Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"client":"0","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475638174Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"client":"1","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475647572Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"client":"2","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475657071Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"io task","client":"2","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475666859Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"io task","client":"0","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475676597Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"io task","client":"1","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"ds_connection connected from Ok(127.0.0.1:55006)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475744734Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"io task","client":"2","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"ds_connection connected from Ok(127.0.0.1:51902)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.47576948Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"io task","client":"0","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"ds_connection connected from Ok(127.0.0.1:50833)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.475900625Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"io task","client":"1","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.476475867Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"client":"2","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.476489245Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"client":"0","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.476499953Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"client":"1","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.476792508Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.47683617Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"[0]R flush_numbers[0..12]: [325, 336, 336, 337, 318, 322, 327, 328, 327, 336, 330, 331]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.47688759Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"downstairs","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"[0]R generation[0..12]: [187, 192, 192, 193, 181, 183, 189, 190, 189, 192, 191, 192]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.476900268Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"downstairs","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.476911846Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"downstairs","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"[1]R flush_numbers[0..12]: [325, 336, 336, 337, 318, 322, 327, 328, 327, 336, 330, 331]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.476922414Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"downstairs","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"[1]R generation[0..12]: [187, 192, 192, 193, 181, 183, 189, 190, 189, 192, 191, 192]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.476933112Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"downstairs","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.47694392Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"downstairs","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"[2]R flush_numbers[0..12]: [325, 336, 336, 337, 318, 322, 327, 328, 327, 336, 330, 331]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.476954568Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"downstairs","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"[2]R generation[0..12]: [187, 192, 192, 193, 181, 183, 189, 190, 189, 192, 191, 192]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.476965176Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"downstairs","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.476976094Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"downstairs","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"Max found gen is 194","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.476987402Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"downstairs","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"Generation requested: 194 >= found:194","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.47699723Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"downstairs","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"Next flush: 338","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.477006568Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"downstairs","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.477016136Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"":"downstairs","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.477026034Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.477034793Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"client":"0","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.477044431Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"client":"1","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.477053949Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"client":"2","":"downstairs","session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.477063787Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.477072766Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"167fe7e2-cb36-4480-b1be-67d136c24da0 is now active with session: 9ce1a9e7-5c21-4708-b4d2-dec756e7637e","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.477082674Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.477091422Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"session_id":"9ce1a9e7-5c21-4708-b4d2-dec756e7637e"} {"msg":"The guest has finished waiting for activation with:194","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.47710054Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909} {"msg":"Activated sub_volume 167fe7e2-cb36-4480-b1be-67d136c24da0","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.477108769Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.477116987Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.477448495Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"task":"crutest"} 01/30 Write block 293 len 4096 data: 17 02/30 Read block 166 len 12288 03/30 Write block 367 len 16384 data: 12 9 11 12 04/30 Write block 402 len 40960 data: 14 14 16 13 14 12 12 7 12 14 05/30 Write block 255 len 28672 data: 17 18 20 22 17 22 22 06/30 Read block 93 len 28672 07/30 Read block 490 len 16384 08/30 Read block 578 len 24576 09/30 Write block 175 len 24576 data: 19 17 18 21 18 15 10/30 Read block 279 len 32768 11/30 Read block 46 len 28672 12/30 Flush 13/30 Write block 267 len 8192 data: 14 13 14/30 Read block 17 len 32768 15/30 Read block 166 len 40960 16/30 Read block 382 len 20480 17/30 Write block 209 len 16384 data: 22 26 22 19 18/30 Read block 347 len 12288 19/30 Read block 262 len 4096 20/30 Read block 437 len 24576 21/30 Read block 219 len 36864 22/30 Read block 499 len 40960 23/30 Read block 173 len 16384 24/30 Flush 25/30 Write block 425 len 12288 data: 10 12 16 26/30 Write block 226 len 40960 data: 9 10 11 12 15 12 15 17 16 15 27/30 Read block 590 len 12288 28/30 Write block 327 len 8192 data: 15 12 29/30 Read block 481 len 8192 30/30 Write block 35 len 28672 data: 18 17 18 18 20 19 20 ---------------------------------------------------------------- Crucible gen:194 GIO:true work queues: Upstairs:0 downstairs:18 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1012 Acked Write 2 Done Done Done false 1013 Acked Read 8 Done Sent Done false 1014 Acked Read 10 Done Sent Done false 1015 Acked Read 5 Done Sent Done false 1016 Acked Write 4 Done Sent Done false 1017 Acked Read 3 Done Sent Done false 1018 Acked Read 1 Done Sent Done false 1019 Acked Read 6 Done Sent Done false 1020 Acked Read 9 Done Sent Done false 1021 Acked Read 10 Done Sent Done false 1022 Acked Read 4 Done Sent Done false 1023 Acked Flush 0 Done Sent Done false 1024 Acked Write 3 Done Sent Done false 1025 Acked Write 10 Done Sent Done false 1026 Acked Read 3 Done Sent Done false 1027 Acked Write 2 Done Sent Done false 1028 Acked Read 2 Done Sent Done false 1029 Acked Write 7 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 17 1 19 Done 17 1 17 35 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1023)) Some(JobId(1011)) Some(JobId(1023)) Downstairs last five completed: 1011 1010 1009 1008 1007 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:03:12.486012461Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:12.486244468Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1909,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 325 325 325 F F F 1 020-039 192 192 192 336 336 336 T F T <--- 2 040-059 192 192 192 336 336 336 T F T <--- 3 060-079 193 193 193 337 337 337 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 183 183 183 322 322 322 F F F 6 120-139 189 189 189 327 327 327 F F F 7 140-159 190 190 190 328 328 328 F F F 8 160-179 194 194 194 338 338 338 F F F 9 180-199 194 194 194 338 338 338 F F F 10 200-219 194 191 194 339 330 339 F F F <--- 11 220-239 192 192 192 331 331 331 T F T <--- 12 240-259 194 194 194 338 338 338 F F F 13 260-279 194 194 194 339 338 339 F T F <--- 14 280-299 194 194 194 338 338 338 F F F 15 300-319 190 190 190 328 328 328 F F F 16 320-339 190 190 190 328 328 328 T F T <--- 17 340-359 190 190 190 328 328 328 F F F 18 360-379 194 194 194 338 338 338 F F F 19 380-399 189 189 189 327 327 327 F F F 20 400-419 194 194 194 338 338 338 F F F 21 420-439 192 192 192 336 336 336 T F T <--- 22 440-459 193 193 193 337 337 337 F F F 23 460-479 190 190 190 329 329 329 F F F 24 480-499 191 191 191 330 330 330 F F F 25 500-519 191 191 191 330 330 330 F F F 26 520-539 189 189 189 327 327 327 F F F 27 540-559 184 184 184 323 323 323 F F F 28 560-579 189 189 189 327 327 327 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 194, Max flush: 339 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 97 Verifying data now No change after verify Loop: 97 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 325 325 325 F F F 1 020-039 195 195 195 340 340 340 F F F 2 040-059 195 195 195 340 340 340 F F F 3 060-079 193 193 193 337 337 337 F F F 4 080-099 181 181 181 318 318 318 F F F 5 100-119 183 183 183 322 322 322 F F F 6 120-139 189 189 189 327 327 327 F F F 7 140-159 190 190 190 328 328 328 F F F 8 160-179 194 194 194 338 338 338 F F F 9 180-199 194 194 194 338 338 338 F F F 10 200-219 194 194 194 339 339 339 F F F 11 220-239 195 195 195 340 340 340 F F F 12 240-259 194 194 194 338 338 338 F F F 13 260-279 194 194 194 339 339 339 F F F 14 280-299 194 194 194 338 338 338 F F F 15 300-319 190 190 190 328 328 328 F F F 16 320-339 195 195 195 340 340 340 F F F 17 340-359 190 190 190 328 328 328 F F F 18 360-379 194 194 194 338 338 338 F F F 19 380-399 189 189 189 327 327 327 F F F 20 400-419 194 194 194 338 338 338 F F F 21 420-439 195 195 195 340 340 340 F F F 22 440-459 193 193 193 337 337 337 F F F 23 460-479 190 190 190 329 329 329 F F F 24 480-499 191 191 191 330 330 330 F F F 25 500-519 191 191 191 330 330 330 F F F 26 520-539 189 189 189 327 327 327 F F F 27 540-559 184 184 184 323 323 323 F F F 28 560-579 189 189 189 327 327 327 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 195, Max flush: 340 Begin loop 98 Downstairs to restart: 0 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.727604968Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"task":"crutest"} {"msg":"Failed to get info from \"http://127.0.0.1:12810\" Communication Error: error sending request for url (http://127.0.0.1:12810/region-info)","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:03:16.734723914Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"task":"crutest"} {"msg":"look at: http://127.0.0.1:12820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.734760457Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.741676492Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.741950711Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.741978156Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.741986454Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.741997802Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917} {"msg":"Crucible 82e1c486-0a1c-4a77-b470-b87561286e86 has session id: 7816fd1d-50e8-4883-8f40-ca44fb152088","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742333889Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"Upstairs opts: Upstairs UUID: 82e1c486-0a1c-4a77-b470-b87561286e86, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742362564Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"Crucible stats registered with UUID: 82e1c486-0a1c-4a77-b470-b87561286e86","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742373662Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:03:16.74238292Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742524853Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"io task","client":"1","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742555488Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"io task","client":"0","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742566615Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"io task","client":"2","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742710239Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"task":"crutest"} {"msg":"The guest has requested activation with gen:196","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742725976Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917} {"msg":"82e1c486-0a1c-4a77-b470-b87561286e86 active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.7427586Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742770767Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"client":"0","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742780585Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"client":"1","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742790184Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"client":"2","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742799602Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"io task","client":"2","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.74280955Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"io task","client":"0","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742819558Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"io task","client":"1","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"ds_connection connected from Ok(127.0.0.1:37073)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742906362Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"io task","client":"0","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"ds_connection connected from Ok(127.0.0.1:39066)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.74291745Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"io task","client":"2","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"ds_connection connected from Ok(127.0.0.1:51851)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.742927778Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"io task","client":"1","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.743720099Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"client":"1","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.743752143Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"client":"0","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.74376895Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"client":"2","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744098279Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744120294Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"[0]R flush_numbers[0..12]: [325, 340, 340, 337, 318, 322, 327, 328, 338, 338, 339, 340]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744134942Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"downstairs","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"[0]R generation[0..12]: [187, 195, 195, 193, 181, 183, 189, 190, 194, 194, 194, 195]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744147479Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"downstairs","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744158537Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"downstairs","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"[1]R flush_numbers[0..12]: [325, 340, 340, 337, 318, 322, 327, 328, 338, 338, 339, 340]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744169165Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"downstairs","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"[1]R generation[0..12]: [187, 195, 195, 193, 181, 183, 189, 190, 194, 194, 194, 195]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744179733Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"downstairs","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744190481Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"downstairs","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"[2]R flush_numbers[0..12]: [325, 340, 340, 337, 318, 322, 327, 328, 338, 338, 339, 340]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744201129Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"downstairs","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"[2]R generation[0..12]: [187, 195, 195, 193, 181, 183, 189, 190, 194, 194, 194, 195]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744211457Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"downstairs","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744222305Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"downstairs","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"Max found gen is 196","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744231634Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"downstairs","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"Generation requested: 196 >= found:196","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744241252Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"downstairs","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"Next flush: 341","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.74425065Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"downstairs","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744260178Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"":"downstairs","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744269826Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744278425Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"client":"0","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744289213Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"client":"1","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744298731Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"client":"2","":"downstairs","session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744308429Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744317488Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"82e1c486-0a1c-4a77-b470-b87561286e86 is now active with session: 7816fd1d-50e8-4883-8f40-ca44fb152088","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744327266Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744335964Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"session_id":"7816fd1d-50e8-4883-8f40-ca44fb152088"} {"msg":"The guest has finished waiting for activation with:196","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744344812Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917} {"msg":"Activated sub_volume 82e1c486-0a1c-4a77-b470-b87561286e86","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744352791Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744360799Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.744705045Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"task":"crutest"} 01/30 Write block 328 len 28672 data: 13 14 18 17 19 15 18 02/30 Write block 571 len 20480 data: 12 11 12 11 12 03/30 Read block 205 len 24576 04/30 Read block 13 len 20480 05/30 Write block 248 len 28672 data: 19 19 16 16 18 16 16 06/30 Write block 365 len 24576 data: 14 13 13 10 12 13 07/30 Write block 40 len 28672 data: 20 21 21 21 18 19 13 08/30 Write block 172 len 4096 data: 13 09/30 Read block 130 len 36864 10/30 Write block 195 len 40960 data: 26 25 20 20 16 18 15 19 16 16 11/30 Read block 359 len 24576 12/30 Write block 282 len 4096 data: 8 13/30 Read block 120 len 28672 14/30 Read block 324 len 12288 15/30 Write block 137 len 28672 data: 18 22 25 21 22 19 16 16/30 Write block 205 len 24576 data: 16 16 16 16 23 27 17/30 Flush 18/30 Write block 232 len 12288 data: 16 18 17 19/30 Write block 234 len 24576 data: 18 16 14 14 14 9 20/30 Read block 505 len 32768 21/30 Write block 185 len 32768 data: 15 16 15 14 17 16 21 23 22/30 Write block 142 len 28672 data: 20 17 16 13 12 13 9 23/30 Write block 79 len 28672 data: 10 10 10 12 9 12 15 24/30 Write block 116 len 40960 data: 11 11 13 11 12 14 12 17 16 16 25/30 Read block 152 len 40960 26/30 Write block 305 len 8192 data: 17 17 27/30 Read block 169 len 24576 28/30 Read block 189 len 40960 29/30 Write block 561 len 36864 data: 15 16 16 12 15 15 15 12 12 30/30 Read block 432 len 28672 ---------------------------------------------------------------- Crucible gen:196 GIO:true work queues: Upstairs:0 downstairs:30 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1000 Acked Write 7 Done Done Done false 1001 Acked Write 5 Sent Done Done false 1002 Acked Read 6 Sent Done Done false 1003 Acked Read 5 Sent Done Done false 1004 Acked Write 7 Sent Done Done false 1005 Acked Write 6 Sent Done Done false 1006 Acked Write 7 Sent Done Done false 1007 Acked Write 1 Sent Done Done false 1008 Acked Read 9 Sent Done Done false 1009 Acked Write 10 Sent Done Done false 1010 Acked Read 6 Sent Done Done false 1011 Acked Write 1 Sent Done Done false 1012 Acked Read 7 Sent Done Done false 1013 Acked Read 3 Sent Done Done false 1014 Acked Write 7 Sent Done Done false 1015 Acked Write 6 Sent Done Done false 1016 Acked Flush 0 Sent Done Done false 1017 Acked Write 3 Sent Done Done false 1018 Acked Write 6 Sent Done Done false 1019 Acked Read 8 Sent Done Done false 1020 Acked Write 8 Sent Done Done false 1021 Acked Write 7 Sent Done Done false 1022 Acked Write 7 Sent Done Done false 1023 Acked Write 10 Sent Done Done false 1024 Acked Read 10 Sent Done Done false 1025 Acked Write 2 Sent Done Done false 1026 Acked Read 6 Sent Done Done false 1027 Acked Read 10 Sent Done Done false 1028 Acked Write 9 Sent Done Done false 1029 Acked Read 7 Sent Sent Done false STATES DS:0 DS:1 DS:2 TOTAL Sent 29 1 0 30 Done 1 29 30 60 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: None Some(JobId(1016)) Some(JobId(1016)) Downstairs last five completed: Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:03:16.75354338Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:16.753734264Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1917,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 325 325 325 F F F 1 020-039 195 195 195 340 340 340 F F F 2 040-059 195 196 196 340 341 341 F F F <--- 3 060-079 193 193 193 337 337 337 F T T <--- 4 080-099 181 181 181 318 318 318 F T T <--- 5 100-119 183 183 183 322 322 322 F T T <--- 6 120-139 189 196 196 327 341 341 F T T <--- 7 140-159 190 196 196 328 341 341 F T T <--- 8 160-179 194 196 196 338 341 341 F F F <--- 9 180-199 194 196 196 338 341 341 F T T <--- 10 200-219 194 196 196 339 341 341 F F F <--- 11 220-239 195 195 195 340 340 340 F T T <--- 12 240-259 194 196 196 338 341 341 F F F <--- 13 260-279 194 194 194 339 339 339 F F F 14 280-299 194 196 196 338 341 341 F F F <--- 15 300-319 190 190 190 328 328 328 F T T <--- 16 320-339 195 196 196 340 341 341 T F F <--- 17 340-359 190 190 190 328 328 328 F F F 18 360-379 194 196 196 338 341 341 F F F <--- 19 380-399 189 189 189 327 327 327 F F F 20 400-419 194 194 194 338 338 338 F F F 21 420-439 195 195 195 340 340 340 F F F 22 440-459 193 193 193 337 337 337 F F F 23 460-479 190 190 190 329 329 329 F F F 24 480-499 191 191 191 330 330 330 F F F 25 500-519 191 191 191 330 330 330 F F F 26 520-539 189 189 189 327 327 327 F F F 27 540-559 184 184 184 323 323 323 F F F 28 560-579 189 196 196 327 341 341 F T T <--- 29 580-599 183 183 183 322 322 322 F F F Max gen: 196, Max flush: 341 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 98 Verifying data now No change after verify Loop: 98 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 325 325 325 F F F 1 020-039 195 195 195 340 340 340 F F F 2 040-059 196 196 196 341 341 341 F F F 3 060-079 197 197 197 342 342 342 F F F 4 080-099 197 197 197 342 342 342 F F F 5 100-119 197 197 197 342 342 342 F F F 6 120-139 197 197 197 342 342 342 F F F 7 140-159 197 197 197 342 342 342 F F F 8 160-179 196 196 196 341 341 341 F F F 9 180-199 197 197 197 342 342 342 F F F 10 200-219 196 196 196 341 341 341 F F F 11 220-239 197 197 197 342 342 342 F F F 12 240-259 196 196 196 341 341 341 F F F 13 260-279 194 194 194 339 339 339 F F F 14 280-299 196 196 196 341 341 341 F F F 15 300-319 197 197 197 342 342 342 F F F 16 320-339 196 196 196 341 341 341 F F F 17 340-359 190 190 190 328 328 328 F F F 18 360-379 196 196 196 341 341 341 F F F 19 380-399 189 189 189 327 327 327 F F F 20 400-419 194 194 194 338 338 338 F F F 21 420-439 195 195 195 340 340 340 F F F 22 440-459 193 193 193 337 337 337 F F F 23 460-479 190 190 190 329 329 329 F F F 24 480-499 191 191 191 330 330 330 F F F 25 500-519 191 191 191 330 330 330 F F F 26 520-539 189 189 189 327 327 327 F F F 27 540-559 184 184 184 323 323 323 F F F 28 560-579 197 197 197 342 342 342 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 197, Max flush: 342 Begin loop 99 Downstairs to restart: 1 {"msg":"look at: http://127.0.0.1:12810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.026174222Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"task":"crutest"} {"msg":"RI is: RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.0344081Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"task":"crutest"} {"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.034693477Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"a81b42ee266cbdd6371f13d7b3f8dd00b79d3cab\",\n git_commit_timestamp: \"2026-07-20T22:41:44.000000000Z\",\n git_branch: \"renovate/crate-rand-vulnerability\",\n rustc_semver: \"1.90.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"1159e78c4747b02ef996e55082b704c09b970588\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.034722712Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925} {"msg":"Upstairs <-> Downstairs Message Version: 13","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.03473158Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 20, shift: 12 }, extent_count: 30, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.034745897Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925} {"msg":"Crucible a3032407-43aa-4acb-8180-7a1d3b5d2fac has session id: 87305609-fd0b-41ed-bc34-77b3c55fd9c3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035071226Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"Upstairs opts: Upstairs UUID: a3032407-43aa-4acb-8180-7a1d3b5d2fac, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035093222Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"Crucible stats registered with UUID: a3032407-43aa-4acb-8180-7a1d3b5d2fac","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.03510299Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"could not find Downstairs address for Nexus","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:03:21.035112159Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035245974Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"io task","client":"1","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.03526272Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"io task","client":"0","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"client is waiting for oneshot","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035272599Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"io task","client":"2","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.03542414Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"task":"crutest"} {"msg":"The guest has requested activation with gen:198","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035440697Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925} {"msg":"a3032407-43aa-4acb-8180-7a1d3b5d2fac active request set","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035458504Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035467332Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"client":"0","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.03547702Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"client":"1","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035486588Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"client":"2","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035496037Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"io task","client":"2","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035507525Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"io task","client":"0","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035517593Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"io task","client":"1","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"ds_connection connected from Ok(127.0.0.1:50350)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035546637Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"io task","client":"2","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"ds_connection connected from Ok(127.0.0.1:48738)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035557695Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"io task","client":"1","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"ds_connection connected from Ok(127.0.0.1:53566)","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.035652337Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"io task","client":"0","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036227Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"client":"0","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036294877Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"client":"1","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036348617Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"client":"2","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036561837Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036594031Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"[0]R flush_numbers[0..12]: [325, 340, 341, 342, 342, 342, 342, 342, 341, 342, 341, 342]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.0366548Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"downstairs","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"[0]R generation[0..12]: [187, 195, 196, 197, 197, 197, 197, 197, 196, 197, 196, 197]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036667257Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"downstairs","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036678285Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"downstairs","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"[1]R flush_numbers[0..12]: [325, 340, 341, 342, 342, 342, 342, 342, 341, 342, 341, 342]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036688933Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"downstairs","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"[1]R generation[0..12]: [187, 195, 196, 197, 197, 197, 197, 197, 196, 197, 196, 197]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036701381Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"downstairs","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036712249Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"downstairs","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"[2]R flush_numbers[0..12]: [325, 340, 341, 342, 342, 342, 342, 342, 341, 342, 341, 342]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036722737Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"downstairs","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"[2]R generation[0..12]: [187, 195, 196, 197, 197, 197, 197, 197, 196, 197, 196, 197]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036733085Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"downstairs","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036743833Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"downstairs","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"Max found gen is 198","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036753241Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"downstairs","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"Generation requested: 198 >= found:198","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036762789Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"downstairs","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"Next flush: 343","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036771978Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"downstairs","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036781476Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"":"downstairs","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036791144Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036799763Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"client":"0","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036809371Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"client":"1","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036818829Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"client":"2","":"downstairs","session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036828417Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036837395Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"a3032407-43aa-4acb-8180-7a1d3b5d2fac is now active with session: 87305609-fd0b-41ed-bc34-77b3c55fd9c3","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036847184Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036857532Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"session_id":"87305609-fd0b-41ed-bc34-77b3c55fd9c3"} {"msg":"The guest has finished waiting for activation with:198","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.03686654Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925} {"msg":"Activated sub_volume a3032407-43aa-4acb-8180-7a1d3b5d2fac","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036874798Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925} {"msg":"Wait for a query_work_queue command to finish before sending IO","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.036883007Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"task":"crutest"} Disk: sv:1 bs:4096 ts:2457600 tb:600 max_io:256 or 1048576 Loading write count information from file "/tmp/test_live_repair/test_repair_verify.data" {"msg":"Run Repair workload","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.037211755Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"task":"crutest"} 01/30 Flush 02/30 Write block 292 len 16384 data: 20 18 15 11 03/30 Write block 239 len 40960 data: 10 14 14 15 17 16 18 18 20 20 04/30 Read block 547 len 32768 05/30 Write block 34 len 28672 data: 19 19 18 19 19 21 21 06/30 Read block 107 len 24576 07/30 Read block 316 len 8192 08/30 Write block 326 len 20480 data: 15 16 14 15 19 09/30 Read block 144 len 4096 10/30 Write block 211 len 40960 data: 23 20 17 20 20 19 19 17 13 15 11/30 Write block 306 len 20480 data: 18 18 17 16 19 12/30 Write block 325 len 24576 data: 16 16 17 15 16 20 13/30 Read block 367 len 28672 14/30 Write block 429 len 40960 data: 15 15 11 9 10 10 12 10 16 14 15/30 Flush 16/30 Write block 49 len 12288 data: 11 9 10 17/30 Read block 269 len 8192 18/30 Write block 521 len 28672 data: 15 16 17 19 20 18 19 19/30 Write block 467 len 20480 data: 8 10 12 13 13 20/30 Write block 528 len 24576 data: 17 15 11 10 10 10 21/30 Flush 22/30 Flush 23/30 Write block 421 len 36864 data: 13 14 11 11 11 13 17 14 16 24/30 Write block 324 len 28672 data: 14 17 17 18 16 17 21 25/30 Read block 532 len 36864 26/30 Write block 56 len 16384 data: 20 18 16 18 27/30 Read block 252 len 12288 28/30 Read block 229 len 20480 29/30 Read block 192 len 16384 30/30 Write block 531 len 8192 data: 11 11 ---------------------------------------------------------------- Crucible gen:198 GIO:true work queues: Upstairs:0 downstairs:29 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 1001 Acked Write 4 Done Done Done false 1002 Acked Write 10 Done Done Done false 1003 Acked Read 8 Done Done Done false 1004 Acked Write 7 Done Sent Done false 1005 Acked Read 6 Done Sent Done false 1006 Acked Read 2 Done Sent Done false 1007 Acked Write 5 Done Sent Done false 1008 Acked Read 1 Done Sent Done false 1009 Acked Write 10 Done Sent Done false 1010 Acked Write 5 Done Sent Done false 1011 Acked Write 6 Done Sent Done false 1012 Acked Read 7 Done Sent Done false 1013 Acked Write 10 Done Sent Done false 1014 Acked Flush 0 Done Sent Done false 1015 Acked Write 3 Done Sent Done false 1016 Acked Read 2 Done Sent Done false 1017 Acked Write 7 Done Sent Done false 1018 Acked Write 5 Done Sent Done false 1019 Acked Write 6 Done Sent Done false 1020 Acked Flush 0 Done Sent Done false 1021 Acked Flush 0 Done Sent Done false 1022 Acked Write 9 Done Sent Done false 1023 Acked Write 7 Done Sent Done false 1024 Acked Read 9 Done Sent Done false 1025 Acked Write 4 Done Sent Done false 1026 Acked Read 3 Done Sent Done false 1027 Acked Read 5 Done Sent Done false 1028 Acked Read 4 Done Sent Done false 1029 Acked Write 2 Sent Sent Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 1 26 1 28 Done 28 3 28 59 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: Some(JobId(1021)) Some(JobId(1000)) Some(JobId(1021)) Downstairs last five completed: 1000 Upstairs last five completed: 1029 1028 1027 1026 1025 {"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2026-07-20T23:03:21.046134394Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925} {"msg":"Wrote out file \"/tmp/test_live_repair/test_repair_verify.data\"","v":0,"name":"crucible","level":30,"time":"2026-07-20T23:03:21.046306812Z","hostname":"w-01KY0VTEM5JBMR6FFPKK5ET1FH","pid":1925,"task":"crutest"} Current downstairs dump with dump args: --no-color -d /var/tmp/test_repair/8810 -d /var/tmp/test_repair/8820 -d /var/tmp/test_repair/8830 EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 325 325 325 F F F 1 020-039 198 195 198 344 340 344 F F F <--- 2 040-059 198 196 198 345 341 345 T F T <--- 3 060-079 197 197 197 342 342 342 F F F 4 080-099 197 197 197 342 342 342 F F F 5 100-119 197 197 197 342 342 342 F F F 6 120-139 197 197 197 342 342 342 F F F 7 140-159 197 197 197 342 342 342 F F F 8 160-179 196 196 196 341 341 341 F F F 9 180-199 197 197 197 342 342 342 F F F 10 200-219 198 196 198 344 341 344 F F F <--- 11 220-239 198 197 198 344 342 344 F T F <--- 12 240-259 198 196 198 344 341 344 F T F <--- 13 260-279 194 194 194 339 339 339 F F F 14 280-299 198 196 198 344 341 344 F T F <--- 15 300-319 198 197 198 344 342 344 F F F <--- 16 320-339 198 196 198 344 341 344 T F T <--- 17 340-359 190 190 190 328 328 328 F F F 18 360-379 196 196 196 341 341 341 F F F 19 380-399 189 189 189 327 327 327 F F F 20 400-419 194 194 194 338 338 338 F F F 21 420-439 198 195 198 344 340 344 T F T <--- 22 440-459 193 193 193 337 337 337 F F F 23 460-479 198 190 198 345 329 345 F F F <--- 24 480-499 191 191 191 330 330 330 F F F 25 500-519 191 191 191 330 330 330 F F F 26 520-539 198 189 198 345 327 345 T F T <--- 27 540-559 184 184 184 323 323 323 F F F 28 560-579 197 197 197 342 342 342 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 198, Max flush: 345 Error: Difference in extent metadata found! Stack backtrace: 0: anyhow::error::::msg at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/backtrace.rs:10:14 1: anyhow::__private::format_err at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.102/src/lib.rs:687:13 2: crucible_downstairs::dump::dump_region at /work/oxidecomputer/crucible/downstairs/src/dump.rs:382:9 A Difference in extent metadata is expected here On loop 99 Verifying data now No change after verify Loop: 99 Downstairs dump after verify (and repair): EXT BLOCKS GEN0 GEN1 GEN2 FL0 FL1 FL2 D0 D1 D2 DIFF 0 000-019 187 187 187 325 325 325 F F F 1 020-039 198 198 198 344 344 344 F F F 2 040-059 199 199 199 346 346 346 F F F 3 060-079 197 197 197 342 342 342 F F F 4 080-099 197 197 197 342 342 342 F F F 5 100-119 197 197 197 342 342 342 F F F 6 120-139 197 197 197 342 342 342 F F F 7 140-159 197 197 197 342 342 342 F F F 8 160-179 196 196 196 341 341 341 F F F 9 180-199 197 197 197 342 342 342 F F F 10 200-219 198 198 198 344 344 344 F F F 11 220-239 198 198 198 344 344 344 F F F 12 240-259 198 198 198 344 344 344 F F F 13 260-279 194 194 194 339 339 339 F F F 14 280-299 198 198 198 344 344 344 F F F 15 300-319 198 198 198 344 344 344 F F F 16 320-339 199 199 199 346 346 346 F F F 17 340-359 190 190 190 328 328 328 F F F 18 360-379 196 196 196 341 341 341 F F F 19 380-399 189 189 189 327 327 327 F F F 20 400-419 194 194 194 338 338 338 F F F 21 420-439 199 199 199 346 346 346 F F F 22 440-459 193 193 193 337 337 337 F F F 23 460-479 198 198 198 345 345 345 F F F 24 480-499 191 191 191 330 330 330 F F F 25 500-519 191 191 191 330 330 330 F F F 26 520-539 199 199 199 346 346 346 F F F 27 540-559 184 184 184 323 323 323 F F F 28 560-579 197 197 197 342 342 342 F F F 29 580-599 183 183 183 322 322 322 F F F Max gen: 199, Max flush: 346 7:28 Test duration Test completed real 7:27.858808987 user 25.515381332 sys 14.237116544 trap 0.019161093 tflt 0.063944658 dflt 0.066912712 kflt 0.004246034 lock 11:57:25.908774768 slp 36:11.840959592 lat 2.336212797 stop 0.019618662